Sponsored Content
Top Forums Shell Programming and Scripting sftp - get newly created files on incremental basis Post 302480382 by ravi.videla on Tuesday 14th of December 2010 06:08:18 PM
Old 12-14-2010
sftp - get newly created files on incremental basis

Hi,

We have a sftp server which creates files daily and keeps 6 months of files on the server. We are creating a daily job to get the files and load into database. My problem is "how to get ONLY those files which got created after my last get". Let me provide some more details to it.

Below example shows files available on server as of day1, 2,3.

Code:
Day1:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip

Day2:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip

Day3:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-11.2010-12-11_19.15.01.zip
customer_data_2010-12-11.2010-12-11_19.15.01.zip
customer_data_2010-12-12.2010-12-13_03.15.01.zip
customer_data_2010-12-12.2010-12-13_03.15.01.zip

To get files, Day1 is easier as we can simply get all 4 files. You can see two more files got created on day2. So we should get only those 2 onto database server. On day3, we see 4 more new files got created. Please let me know if you think of any easy way to get files from sftp server that are created after last get. Appreciate your help on this.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to FTP all newly created but the current open file?

An application running on HP-UX constantly generates new text log files ( I think using logpipe ). Any new file created requires to be ftp'ed to an offline server, however I want to make sure that the current file being written should not be transferred. For examples consider the following files... (3 Replies)
Discussion started by: indianya
3 Replies

2. UNIX for Dummies Questions & Answers

Newly created files default group and write permissions

Whenever I create a new file the group name is "dnn" and the file permissions are "-rw-r--r--". How do I get it so when I create files (with vi or other programs) that the default group is "sss" and the permissions are 770? (I am running HP-UNIX) Thanks, GoldFish (2 Replies)
Discussion started by: goldfish
2 Replies

3. UNIX for Dummies Questions & Answers

Forcing UID on Files/Dirs Created with SFTP?

I have a situation where I have to provide Windows based users with access to specific files and directories on a *nix web mail server. The users cannot use the CLI, so SSH is out. They've previously used a product called Webdrive to access *nix boxes via SFTP. The files and dirs they need... (5 Replies)
Discussion started by: deckard
5 Replies

4. UNIX for Dummies Questions & Answers

Help with command to find all newly created files in a given time period

Hi all! Can someone please help me create a command to accomplish the following task. I have a parent directory called ex. /var/www/parent and it has a bunch of sub-directories called /var/www/parent/1, var/www/parent/1/xyz/ and etc. What I would like to do is to count the number of files... (2 Replies)
Discussion started by: bbzor
2 Replies

5. Shell Programming and Scripting

How to find the newly created directory

Hi, I need to create new directory by increasing the number by 1 of extracted lastly created directory. e.g. Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ become Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ Log\out_log_4\ Can anyone help how to do it in c-shell... (3 Replies)
Discussion started by: Andre_2008
3 Replies

6. UNIX for Advanced & Expert Users

default size of a newly created folder

Hi all, In linux how to create a directory with specified size, so that it can be used only up to the mentioned size. Actually my question is, whether we can do directory quota in linux. mounting the directory in a partiton will do that, but do we have any other option... (1 Reply)
Discussion started by: anishkumarv
1 Replies

7. Solaris

Can't see Newly created LUN by SAN admin

hello, i am an oracle DBA and trying to scan a newly created LUN of 200 GB on fiber channel by SAN admin.we have solaris 10 and SANtoolkit is installed.i tried following to get the new LUN at my machine. go /opt/Netapp/Santoolkit/bin and then ./sanlun lun show but i see only the existing... (12 Replies)
Discussion started by: janakors
12 Replies

8. Shell Programming and Scripting

Copy an array to a newly created directory

hello everyone, I am new to perl script and trying to develop a script as follows. I am trying to Create an array for storing all file names. I am trying to copy $libs into "scratch". however i am unable to do so. Please suggest.. #!/usr/bin/perl use File::Copy; #use... (5 Replies)
Discussion started by: Rashid Khan
5 Replies

9. Hardware

Formatting a newly created lun

Hi , I have created one new lun in my SAN storage and make it visible to my HP servers , but the fdisk -l output is somehow confusing. Do not know what to do next ---------- fdisk -l /dev/sdo1 Disk /dev/sdo1 (Sun disk label): 64 heads, 32 sectors, 10238 cylinders Units =... (7 Replies)
Discussion started by: mishra.sankar
7 Replies

10. AIX

No iscsi available in newly created AIX wpar

AIX 7.1 New to WPAR, hopefully just missing something simple here. Creating the WPAR like this..... (The box where the WPAR is hosted does have an iscsi protocol device) mkwpar -h wpar08 -l -n wpar08 -N interface=en0 address=xxx.xx.xx.xxx netmask=255.255.255.0 -D devname=/dev/iscsi0 -D... (0 Replies)
Discussion started by: TomR
0 Replies
ZIPMERGE(1)						      General Commands Manual						       ZIPMERGE(1)

NAME
zipmerge - merge zip archives SYNOPSIS
zipmerge [-DhIiSsV] target-zip source-zip Op source-zip ... DESCRIPTION
zipmerge merges the source zip archives source-zip into the target zip archive target-zip. By default, files in the source zip archives overwrite existing files of the same name in the target zip archive. Supported options: -D Ignore directory components in file name comparisons. -h Display a short help message and exit. -I Ignore case in file name comparisons -i Ask before overwriting files. See also -s. -S Do not overwrite files that have the same size and CRC32 in both the source and target archives. -s When -i is given, do not before overwriting files that have the same size and CRC32. -V Display version information and exit. EXIT STATUS
zipmerge exits 0 on success and 1 if an error occurred. SEE ALSO
zipcmp(1), ziptorrent(1), libzip(3) AUTHORS
Dieter Baron <dillo@giga.or.at> and Thomas Klausner <tk@giga.or.at> NiH June 4, 2008 ZIPMERGE(1)
All times are GMT -4. The time now is 09:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy