Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

sftp - get newly created files on incremental basis

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sftp - get newly created files on incremental basis
# 1  
Old 12-14-2010
sftp - get newly created files on incremental basis
Hi,

We have a sftp server which creates files daily and keeps 6 months of files on the server. We are creating a daily job to get the files and load into database. My problem is "how to get ONLY those files which got created after my last get". Let me provide some more details to it.

Below example shows files available on server as of day1, 2,3.

Code:
Day1:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip

Day2:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip

Day3:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-11.2010-12-11_19.15.01.zip
customer_data_2010-12-11.2010-12-11_19.15.01.zip
customer_data_2010-12-12.2010-12-13_03.15.01.zip
customer_data_2010-12-12.2010-12-13_03.15.01.zip

To get files, Day1 is easier as we can simply get all 4 files. You can see two more files got created on day2. So we should get only those 2 onto database server. On day3, we see 4 more new files got created. Please let me know if you think of any easy way to get files from sftp server that are created after last get. Appreciate your help on this.
# 2  
Old 12-14-2010
Here is what I can think of..
Maintain a file say "processed_file" which keeps names of all files processed so far. This file will be empty on very 1st day and new file names will get appended to it later on while each run.
Now daily job script will create the list of ALL files available on sftp server (put this file list in some temp file). Get a diff of temp file and processed_file and put the diff result in a new file "TO_BE_PROCESSED" file.
Now get all the files listed in "TO_BE_PROCESSED" file,process them as need and append sftp'd file name in processed_file.
# 3  
Old 12-14-2010
Thanks Anurag. Approach looks OK.
I am thinking more like finding all files which got created on ftp server since our last get "based on timestamp". Any other thoughts?
# 4  
Old 12-15-2010
rsync
# 5  
Old 12-15-2010
We do not have rsync installed on our servers. Any other thoughts from anyone else. Appreciate your help.
# 6  
Old 12-15-2010
How about moving the files to a "processed" sub-directory on the sftp server?
# 7  
Old 12-15-2010
I thought about this option. But could not find any command for moving files in sftp help. mv is not working in sftp prompt.

I also see one limitation with this approach. Let me explain.

Say there are 10 files on ftp server. We will get them using mget call*. while getting the files, may be two more got added. After get is completed, we will do move call* to processed directory. So we would miss two files and never get them for processing. Am i missing something?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

No iscsi available in newly created AIX wpar

AIX 7.1 New to WPAR, hopefully just missing something simple here. Creating the WPAR like this..... (The box where the WPAR is hosted does have an iscsi protocol device) mkwpar -h wpar08 -l -n wpar08 -N interface=en0 address=xxx.xx.xx.xxx netmask=255.255.255.0 -D devname=/dev/iscsi0 -D... (0 Replies)
Discussion started by: TomR
0 Replies

2. Hardware

Formatting a newly created lun

Hi , I have created one new lun in my SAN storage and make it visible to my HP servers , but the fdisk -l output is somehow confusing. Do not know what to do next ---------- fdisk -l /dev/sdo1 Disk /dev/sdo1 (Sun disk label): 64 heads, 32 sectors, 10238 cylinders Units =... (7 Replies)
Discussion started by: mishra.sankar
7 Replies

3. Shell Programming and Scripting

Copy an array to a newly created directory

hello everyone, I am new to perl script and trying to develop a script as follows. I am trying to Create an array for storing all file names. I am trying to copy $libs into "scratch". however i am unable to do so. Please suggest.. #!/usr/bin/perl use File::Copy; #use... (5 Replies)
Discussion started by: Rashid Khan
5 Replies

4. Solaris

Can't see Newly created LUN by SAN admin

hello, i am an oracle DBA and trying to scan a newly created LUN of 200 GB on fiber channel by SAN admin.we have solaris 10 and SANtoolkit is installed.i tried following to get the new LUN at my machine. go /opt/Netapp/Santoolkit/bin and then ./sanlun lun show but i see only the existing... (12 Replies)
Discussion started by: janakors
12 Replies

5. UNIX for Advanced & Expert Users

default size of a newly created folder

Hi all, In linux how to create a directory with specified size, so that it can be used only up to the mentioned size. Actually my question is, whether we can do directory quota in linux. mounting the directory in a partiton will do that, but do we have any other option... (1 Reply)
Discussion started by: anishkumarv
1 Replies

6. Shell Programming and Scripting

How to find the newly created directory

Hi, I need to create new directory by increasing the number by 1 of extracted lastly created directory. e.g. Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ become Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ Log\out_log_4\ Can anyone help how to do it in c-shell... (3 Replies)
Discussion started by: Andre_2008
3 Replies

7. UNIX for Dummies Questions & Answers

Help with command to find all newly created files in a given time period

Hi all! Can someone please help me create a command to accomplish the following task. I have a parent directory called ex. /var/www/parent and it has a bunch of sub-directories called /var/www/parent/1, var/www/parent/1/xyz/ and etc. What I would like to do is to count the number of files... (2 Replies)
Discussion started by: bbzor
2 Replies

8. UNIX for Dummies Questions & Answers

Forcing UID on Files/Dirs Created with SFTP?

I have a situation where I have to provide Windows based users with access to specific files and directories on a *nix web mail server. The users cannot use the CLI, so SSH is out. They've previously used a product called Webdrive to access *nix boxes via SFTP. The files and dirs they need... (5 Replies)
Discussion started by: deckard
5 Replies

9. UNIX for Dummies Questions & Answers

Newly created files default group and write permissions

Whenever I create a new file the group name is "dnn" and the file permissions are "-rw-r--r--". How do I get it so when I create files (with vi or other programs) that the default group is "sss" and the permissions are 770? (I am running HP-UNIX) Thanks, GoldFish (2 Replies)
Discussion started by: goldfish
2 Replies

10. UNIX for Advanced & Expert Users

How to FTP all newly created but the current open file?

An application running on HP-UX constantly generates new text log files ( I think using logpipe ). Any new file created requires to be ftp'ed to an offline server, however I want to make sure that the current file being written should not be transferred. For examples consider the following files... (3 Replies)
Discussion started by: indianya
3 Replies
Login or Register to Ask a Question