Sponsored Content
Top Forums Shell Programming and Scripting sftp - get newly created files on incremental basis Post 302480382 by ravi.videla on Tuesday 14th of December 2010 06:08:18 PM
Old 12-14-2010
sftp - get newly created files on incremental basis

Hi,

We have a sftp server which creates files daily and keeps 6 months of files on the server. We are creating a daily job to get the files and load into database. My problem is "how to get ONLY those files which got created after my last get". Let me provide some more details to it.

Below example shows files available on server as of day1, 2,3.

Code:
Day1:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip

Day2:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip

Day3:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-11.2010-12-11_19.15.01.zip
customer_data_2010-12-11.2010-12-11_19.15.01.zip
customer_data_2010-12-12.2010-12-13_03.15.01.zip
customer_data_2010-12-12.2010-12-13_03.15.01.zip

To get files, Day1 is easier as we can simply get all 4 files. You can see two more files got created on day2. So we should get only those 2 onto database server. On day3, we see 4 more new files got created. Please let me know if you think of any easy way to get files from sftp server that are created after last get. Appreciate your help on this.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to FTP all newly created but the current open file?

An application running on HP-UX constantly generates new text log files ( I think using logpipe ). Any new file created requires to be ftp'ed to an offline server, however I want to make sure that the current file being written should not be transferred. For examples consider the following files... (3 Replies)
Discussion started by: indianya
3 Replies

2. UNIX for Dummies Questions & Answers

Newly created files default group and write permissions

Whenever I create a new file the group name is "dnn" and the file permissions are "-rw-r--r--". How do I get it so when I create files (with vi or other programs) that the default group is "sss" and the permissions are 770? (I am running HP-UNIX) Thanks, GoldFish (2 Replies)
Discussion started by: goldfish
2 Replies

3. UNIX for Dummies Questions & Answers

Forcing UID on Files/Dirs Created with SFTP?

I have a situation where I have to provide Windows based users with access to specific files and directories on a *nix web mail server. The users cannot use the CLI, so SSH is out. They've previously used a product called Webdrive to access *nix boxes via SFTP. The files and dirs they need... (5 Replies)
Discussion started by: deckard
5 Replies

4. UNIX for Dummies Questions & Answers

Help with command to find all newly created files in a given time period

Hi all! Can someone please help me create a command to accomplish the following task. I have a parent directory called ex. /var/www/parent and it has a bunch of sub-directories called /var/www/parent/1, var/www/parent/1/xyz/ and etc. What I would like to do is to count the number of files... (2 Replies)
Discussion started by: bbzor
2 Replies

5. Shell Programming and Scripting

How to find the newly created directory

Hi, I need to create new directory by increasing the number by 1 of extracted lastly created directory. e.g. Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ become Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ Log\out_log_4\ Can anyone help how to do it in c-shell... (3 Replies)
Discussion started by: Andre_2008
3 Replies

6. UNIX for Advanced & Expert Users

default size of a newly created folder

Hi all, In linux how to create a directory with specified size, so that it can be used only up to the mentioned size. Actually my question is, whether we can do directory quota in linux. mounting the directory in a partiton will do that, but do we have any other option... (1 Reply)
Discussion started by: anishkumarv
1 Replies

7. Solaris

Can't see Newly created LUN by SAN admin

hello, i am an oracle DBA and trying to scan a newly created LUN of 200 GB on fiber channel by SAN admin.we have solaris 10 and SANtoolkit is installed.i tried following to get the new LUN at my machine. go /opt/Netapp/Santoolkit/bin and then ./sanlun lun show but i see only the existing... (12 Replies)
Discussion started by: janakors
12 Replies

8. Shell Programming and Scripting

Copy an array to a newly created directory

hello everyone, I am new to perl script and trying to develop a script as follows. I am trying to Create an array for storing all file names. I am trying to copy $libs into "scratch". however i am unable to do so. Please suggest.. #!/usr/bin/perl use File::Copy; #use... (5 Replies)
Discussion started by: Rashid Khan
5 Replies

9. Hardware

Formatting a newly created lun

Hi , I have created one new lun in my SAN storage and make it visible to my HP servers , but the fdisk -l output is somehow confusing. Do not know what to do next ---------- fdisk -l /dev/sdo1 Disk /dev/sdo1 (Sun disk label): 64 heads, 32 sectors, 10238 cylinders Units =... (7 Replies)
Discussion started by: mishra.sankar
7 Replies

10. AIX

No iscsi available in newly created AIX wpar

AIX 7.1 New to WPAR, hopefully just missing something simple here. Creating the WPAR like this..... (The box where the WPAR is hosted does have an iscsi protocol device) mkwpar -h wpar08 -l -n wpar08 -N interface=en0 address=xxx.xx.xx.xxx netmask=255.255.255.0 -D devname=/dev/iscsi0 -D... (0 Replies)
Discussion started by: TomR
0 Replies
zipnote(1)						      General Commands Manual							zipnote(1)

NAME
zipsplit - split a zipfile into smaller zipfiles SYNOPSIS
zipsplit [-t] [-i] [-p] [-s] [-n size] [-r room] [-b path] [-h] [-v] [-L] zipfile ARGUMENTS
zipfile Zipfile to split. OPTIONS
-t Report how many files it will take, but don't make them. -i Make index (zipsplit.idx) and count its size against first zip file. -n size Make zip files no larger than "size" (default = 36000). -r room Leave room for "room" bytes on the first disk (default = 0). -b path Use path for the output zip files. -p Pause between output zip files. -s Do a sequential split even if it takes more zip files. -h Show a short help. -v Show version information. -L Show software license. DESCRIPTION
zipsplit reads a zipfile and splits it into smaller zipfiles. EXAMPLES
To be filled in. BUGS
Does not yet support large (> 2 GB) or split archives. SEE ALSO
zip(1), unzip(1) AUTHOR
Info-ZIP v3.0 of 8 May 2008 zipnote(1)
All times are GMT -4. The time now is 03:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy