sftp - get newly created files on incremental basis


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sftp - get newly created files on incremental basis
# 8  
Old 12-15-2010
Yes mget is a bad way to go.
You are better of doing a ls of files first and then generate a script (from the filelist) that gets and moves each file individually without using wildcards.

Fetch the file to a .tfr file first and shell out a mv command to move to final name (stops any local scripts picking up files still being transfered).

Here is a quick script to get you going (no exit status checking or log file writing)

Code:
cat > /tmp/list.tmp$$ <<EOF
cd src_dir
ls -1
quit
EOF
# Remove all <CTRL-M> chars from filename
# Ignore any files where name:
#     begins with . (dot) or / (slash)
#     contains a space
#
sed -e '/ /d' -e '/^\./d' -e '/^\//d' -e 's/^M//g' /tmp/list.tmp$$ > /tmp/list.cmd$$
sftp -b /tmp/list.cmd$$ user@host > /tmp/filelist
echo "cd src_dir" > /tmp/get.cmd$$
for file in `cat /tmp/filelist`
do
    echo "get $file $file.tfr" >> /tmp/get.cmd$$
    echo "!mv $file.tfr $file" >> /tmp/get.cmd$$
    echo "rename $file ../processed/$file" >> /tmp/get.cmd$$
done
echo "quit" >> /tmp/get.cmd$$
sftp -b /tmp/get.cmd$$ user@host > /tmp/filelist
rm /tmp/list.tmp$$ /tmp/list.cmd$$ /tmp/get.cmd$$

Note ^M represents a <CR> character type CTRL-V CTRL-M to enter in vi.
This User Gave Thanks to Chubler_XL For This Post:
# 9  
Old 12-17-2010
Thanks Chubler for sample script. It works fine for my problem.
Only one issue i can think of is how to ignore files which are incomplete or in the process of creation. ls -1 is giving list of all files present. Is there a way to skip those files by any way? Please let me know.
# 10  
Old 12-18-2010
maybe use the find utility to only move files that are more than an hour old?
# 11  
Old 12-18-2010
find command does not work from sftp prompt. It is not list of ftp commands. Am i missing something?
# 12  
Old 12-18-2010
Yes, you are missing something. The find command is a separate utility and you need to modify the script supplied by Chubler_XL to utilize it. Man find(1) for more information. It is not hard to do.
# 13  
Old 12-19-2010
fpmurphy, ls is supported via sftp, but find isn't I suspect that ravi.videla dosn't have shell (ssh) access to sftp server so probably can run find there.


ravi.videla, do you have any control of the creation process, if so a couple of suggestions:
  1. upload/build files in another directory (on the same filesystem) and then rename (mv) them after the upload/creation is done.
  2. upload/build with different name (eg .trf extension) and rename after completed. You then exclude .tfr files with the sed:
    Code:
    sed -e '/ /d' -e '/^\./d' -e '/*.tfr/d' -e '/^\//d' -e 's/^M//g' /tmp/list.tmp$$ > /tmp/list.cmd$$

# 14  
Old 12-20-2010
I am getting "invalid command" error for fpmurphy in sftp prompt.
I do not have any control on the file creation process. All we have is ftp server user/pass, directory from where we should get files. It looks like, I should request source team use a different file extension like .trf when they are creating files and name it with proper extension after file is completely created.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

No iscsi available in newly created AIX wpar

AIX 7.1 New to WPAR, hopefully just missing something simple here. Creating the WPAR like this..... (The box where the WPAR is hosted does have an iscsi protocol device) mkwpar -h wpar08 -l -n wpar08 -N interface=en0 address=xxx.xx.xx.xxx netmask=255.255.255.0 -D devname=/dev/iscsi0 -D... (0 Replies)
Discussion started by: TomR
0 Replies

2. Hardware

Formatting a newly created lun

Hi , I have created one new lun in my SAN storage and make it visible to my HP servers , but the fdisk -l output is somehow confusing. Do not know what to do next ---------- fdisk -l /dev/sdo1 Disk /dev/sdo1 (Sun disk label): 64 heads, 32 sectors, 10238 cylinders Units =... (7 Replies)
Discussion started by: mishra.sankar
7 Replies

3. Shell Programming and Scripting

Copy an array to a newly created directory

hello everyone, I am new to perl script and trying to develop a script as follows. I am trying to Create an array for storing all file names. I am trying to copy $libs into "scratch". however i am unable to do so. Please suggest.. #!/usr/bin/perl use File::Copy; #use... (5 Replies)
Discussion started by: Rashid Khan
5 Replies

4. Solaris

Can't see Newly created LUN by SAN admin

hello, i am an oracle DBA and trying to scan a newly created LUN of 200 GB on fiber channel by SAN admin.we have solaris 10 and SANtoolkit is installed.i tried following to get the new LUN at my machine. go /opt/Netapp/Santoolkit/bin and then ./sanlun lun show but i see only the existing... (12 Replies)
Discussion started by: janakors
12 Replies

5. UNIX for Advanced & Expert Users

default size of a newly created folder

Hi all, In linux how to create a directory with specified size, so that it can be used only up to the mentioned size. Actually my question is, whether we can do directory quota in linux. mounting the directory in a partiton will do that, but do we have any other option... (1 Reply)
Discussion started by: anishkumarv
1 Replies

6. Shell Programming and Scripting

How to find the newly created directory

Hi, I need to create new directory by increasing the number by 1 of extracted lastly created directory. e.g. Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ become Log\out_log_1\ Log\out_log_2\ Log\out_log_3\ Log\out_log_4\ Can anyone help how to do it in c-shell... (3 Replies)
Discussion started by: Andre_2008
3 Replies

7. UNIX for Dummies Questions & Answers

Help with command to find all newly created files in a given time period

Hi all! Can someone please help me create a command to accomplish the following task. I have a parent directory called ex. /var/www/parent and it has a bunch of sub-directories called /var/www/parent/1, var/www/parent/1/xyz/ and etc. What I would like to do is to count the number of files... (2 Replies)
Discussion started by: bbzor
2 Replies

8. UNIX for Dummies Questions & Answers

Forcing UID on Files/Dirs Created with SFTP?

I have a situation where I have to provide Windows based users with access to specific files and directories on a *nix web mail server. The users cannot use the CLI, so SSH is out. They've previously used a product called Webdrive to access *nix boxes via SFTP. The files and dirs they need... (5 Replies)
Discussion started by: deckard
5 Replies

9. UNIX for Dummies Questions & Answers

Newly created files default group and write permissions

Whenever I create a new file the group name is "dnn" and the file permissions are "-rw-r--r--". How do I get it so when I create files (with vi or other programs) that the default group is "sss" and the permissions are 770? (I am running HP-UNIX) Thanks, GoldFish (2 Replies)
Discussion started by: goldfish
2 Replies

10. UNIX for Advanced & Expert Users

How to FTP all newly created but the current open file?

An application running on HP-UX constantly generates new text log files ( I think using logpipe ). Any new file created requires to be ftp'ed to an offline server, however I want to make sure that the current file being written should not be transferred. For examples consider the following files... (3 Replies)
Discussion started by: indianya
3 Replies
Login or Register to Ask a Question