Sponsored Content
Top Forums Shell Programming and Scripting sftp - get newly created files on incremental basis Post 302480382 by ravi.videla on Tuesday 14th of December 2010 06:08:18 PM
Old 12-14-2010
sftp - get newly created files on incremental basis

Hi,

We have a sftp server which creates files daily and keeps 6 months of files on the server. We are creating a daily job to get the files and load into database. My problem is "how to get ONLY those files which got created after my last get". Let me provide some more details to it.

Below example shows files available on server as of day1, 2,3.

Code:
Day1:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip

Day2:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip

Day3:
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-09.2010-12-10_03.15.01.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-10_19.15.02.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-10.2010-12-11_03.15.01.zip
customer_data_2010-12-11.2010-12-11_19.15.01.zip
customer_data_2010-12-11.2010-12-11_19.15.01.zip
customer_data_2010-12-12.2010-12-13_03.15.01.zip
customer_data_2010-12-12.2010-12-13_03.15.01.zip

To get files, Day1 is easier as we can simply get all 4 files. You can see two more files got created on day2. So we should get only those 2 onto database server. On day3, we see 4 more new files got created. Please let me know if you think of any easy way to get files from sftp server that are created after last get. Appreciate your help on this.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to FTP all newly created but the current open file?

An application running on HP-UX constantly generates new text log files ( I think using logpipe ). Any new file created requires to be ftp'ed to an offline server, however I want to make sure that the current file being written should not be transferred. For examples consider the following files... (3 Replies)
Discussion started by: indianya
3 Replies

2. UNIX for Dummies Questions & Answers

Newly created files default group and write permissions

Whenever I create a new file the group name is "dnn" and the file permissions are "-rw-r--r--". How do I get it so when I create files (with vi or other programs) that the default group is "sss" and the permissions are 770? (I am running HP-UNIX) Thanks, GoldFish (2 Replies)
Discussion started by: goldfish
2 Replies

3. UNIX for Dummies Questions & Answers

Forcing UID on Files/Dirs Created with SFTP?

I have a situation where I have to provide Windows based users with access to specific files and directories on a *nix web mail server. The users cannot use the CLI, so SSH is out. They've previously used a product called Webdrive to access *nix boxes via SFTP. The files and dirs they need... (5 Replies)
Discussion started by: deckard
5 Replies

4. UNIX for Dummies Questions & Answers

Help with command to find all newly created files in a given time period

Hi all! Can someone please help me create a command to accomplish the following task. I have a parent directory called ex. /var/www/parent and it has a bunch of sub-directories called /var/www/parent/1, var/www/parent/1/xyz/ and etc. What I would like to do is to count the number of files... (2 Replies)
Discussion started by: bbzor
2 Replies

5. Shell Programming and Scripting

How to find the newly created directory

Hi, I need to create new directory by increasing the number by 1 of extracted lastly created directory. e.g. Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ become Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ Log\out_log_4\ Can anyone help how to do it in c-shell... (3 Replies)
Discussion started by: Andre_2008
3 Replies

6. UNIX for Advanced & Expert Users

default size of a newly created folder

Hi all, In linux how to create a directory with specified size, so that it can be used only up to the mentioned size. Actually my question is, whether we can do directory quota in linux. mounting the directory in a partiton will do that, but do we have any other option... (1 Reply)
Discussion started by: anishkumarv
1 Replies

7. Solaris

Can't see Newly created LUN by SAN admin

hello, i am an oracle DBA and trying to scan a newly created LUN of 200 GB on fiber channel by SAN admin.we have solaris 10 and SANtoolkit is installed.i tried following to get the new LUN at my machine. go /opt/Netapp/Santoolkit/bin and then ./sanlun lun show but i see only the existing... (12 Replies)
Discussion started by: janakors
12 Replies

8. Shell Programming and Scripting

Copy an array to a newly created directory

hello everyone, I am new to perl script and trying to develop a script as follows. I am trying to Create an array for storing all file names. I am trying to copy $libs into "scratch". however i am unable to do so. Please suggest.. #!/usr/bin/perl use File::Copy; #use... (5 Replies)
Discussion started by: Rashid Khan
5 Replies

9. Hardware

Formatting a newly created lun

Hi , I have created one new lun in my SAN storage and make it visible to my HP servers , but the fdisk -l output is somehow confusing. Do not know what to do next ---------- fdisk -l /dev/sdo1 Disk /dev/sdo1 (Sun disk label): 64 heads, 32 sectors, 10238 cylinders Units =... (7 Replies)
Discussion started by: mishra.sankar
7 Replies

10. AIX

No iscsi available in newly created AIX wpar

AIX 7.1 New to WPAR, hopefully just missing something simple here. Creating the WPAR like this..... (The box where the WPAR is hosted does have an iscsi protocol device) mkwpar -h wpar08 -l -n wpar08 -N interface=en0 address=xxx.xx.xx.xxx netmask=255.255.255.0 -D devname=/dev/iscsi0 -D... (0 Replies)
Discussion started by: TomR
0 Replies
ZIP_SET_DEFAULT_PASSWORD(3)				     Library Functions Manual				       ZIP_SET_DEFAULT_PASSWORD(3)

NAME
zip_set_default_password - set default password for encrypted files in zip LIBRARY
libzip (-lzip) SYNOPSIS
#include <zip.h> int zip_set_default_password(struct zip *archive, const char *password); DESCRIPTION
The zip_set_default_password function sets the default password used when accessing encrypted files. If password is NULL, the default password is unset. If you prefer a different password for single files, use zip_fopen_encrypted(3) instead of zip_fopen(3). Usually, however, the same pass- word is used for every file in an zip archive. RETURN VALUES
Upon successful completion 0 is returned. Otherwise, -1 is returned and the error information in archive is set to indicate the error. ERRORS
zip_set_default_password fails if: [ZIP_ER_MEMORY] Required memory could not be allocated. SEE ALSO
libzip(3), zip_fopen(3), zip_fopen_encrypted(3) AUTHORS
Dieter Baron <dillo@giga.or.at> and Thomas Klausner <tk@giga.or.at> NiH January 3, 2011 ZIP_SET_DEFAULT_PASSWORD(3)
All times are GMT -4. The time now is 11:03 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy