Need to automate sftp and always get the latest file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need to automate sftp and always get the latest file
# 1  
Old 11-12-2014
Need to automate sftp and always get the latest file

hi guys the script is working but i need to change the file that i want to get to the latest.. i will use this as cron so i can get the latest sftp file evryday.
pls help..

Code:
set user "big_user"
set IP "XX.XX.XX"
set dir "/export/home/oracle/REF/INPUT"

spawn sftp $user@$IP
expect sftp>

send "cd $dir\r"
expect sftp>

send "get /export/home/oracle/REF/INPUT/3G_20141112*.csv\r"
expect sftp>

send "exit\r"
expect eof

Moderator's Comments:
Mod Comment Please use code tags next time for your code and data. Thanks

Last edited by vbe; 11-12-2014 at 05:20 AM..
# 2  
Old 11-12-2014
Can you generate and exchange SSH keys? It will make the whole thing much simpler. Eventually, you probably need a three step process:-
  1. SFTP list all the relevant remote files
  2. Determine the file you want to get
  3. SFTP get the file

If you exchange SSH keys then the SFTP steps become much easier.



Robin
This User Gave Thanks to rbatte1 For This Post:
# 3  
Old 11-13-2014
thanks rob

all the files are .csv's. i just need to get the latest file. and cron it to automate daily reports.
# 4  
Old 11-13-2014
Hello Jonny

So can you follow the logic for each of the steps and get a working solution? If the approach is acceptable, let us know how far you are getting on each step and what you have tried. It's easier to work with your code so we can suggest adjustments in a way that you will understand and be able to support for your future. It will also show us what tools you favour so we can meet that too.


Regards,
Robin
# 5  
Old 11-13-2014
Hi Sir Rob

the script above is what i am using,
can u help enhance it?
i dont really know what to do..

thanks so much
# 6  
Old 11-14-2014
For small and few files, mget them all and delete the unwanted locally.
# 7  
Old 11-14-2014
Well, it would be polite to use a correct name when it's given. I'm Robin.

If you want to get one file only rather than to get them all and delete the unwanted, the command dir within sftp will help you list them out. Capture the output of sftp to a file and work with that. If the file name structure is sufficient (you seem to have a name that includes YYYY MM DD, which is great) then that makes it easy.

As an alternate, try ls -lrt within sftp and see if the remote server will list them newest-last for you. I don't think it's always supported though. You can then trim the output to get the file name you need back in normal shell script and then pass this name into a second sftp to get the required file.

Something like:-
Code:
#!/bin/ksh
sftp user@host <<-EOSFTP >/tmp/sftp.listlog
cd /export/home/oracle/REF/INPUT
ls -lrt 3G_*
EOSFTP

tail -1 /tmp/sftp.listlog | read perm links owner group size dte1 dte2 dte3 file

sftp user@host <<-EOSFTP >/tmp/sftp.xferlog
cd /export/home/oracle/REF/INPUT
get $file
EOSFTP

Error checking would be sensible, but this will prompt, so exchange of SSH keys would be needed to automate it. Using expect might get you round it, but it is not the preferred way.

Do you know how to generate & exchange SSH keys? We'd need to know which OS the local & remote servers are running.



Regards,
Robin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Retrieve the Latest file in a folder using SFTP

HI Guys, Can anyone help me to retrieve the latest file from below example using SFTP I have below list of files in a folder v403t.lstprgms.sortin1 val027.d099.fwest.oct2711 xcelrptd.d1400sqp.dec1713.t040459.@02del xcelrptd.d1400sqp.dec1713.t073308.@02del... (3 Replies)
Discussion started by: heye18
3 Replies

2. Shell Programming and Scripting

Sftp automate

hi, I am trying to automate a file download process using sftp. There is some logic to download files. 1) I need to login to destination server and then go to folder. 2) find list of files and count 3) using list of files I need to eliminate three selective files and download remaining... (1 Reply)
Discussion started by: getmilo
1 Replies

3. UNIX for Dummies Questions & Answers

Find the latest file on remote sftp

Hi All, I need your help in finding the latest files in remote sftp and get those files to local server and process them. Please let me know I appreciate your valuable inputs. Thanks raj (7 Replies)
Discussion started by: rajeevm
7 Replies

4. Shell Programming and Scripting

Using expect to automate sftp

I am trying to use a for loop in my expect cmdFile that I am calling. I want to be able to call either one file name or a series of file names in the working directory (that I won't know the names before hand) and then pass the names to the sftp program. Something like for i in (ls *txt) do (0 Replies)
Discussion started by: vedder191
0 Replies

5. Shell Programming and Scripting

Need to copy latest file using SFTP

Hi All, In my unix server, I have the following files: h1.txt h2.txt h3.txt and through SFTP i need to copy only the latest file to another unix server. Can you please let me know what command i need to use. Thanks in Advance, (2 Replies)
Discussion started by: HemaV
2 Replies

6. Shell Programming and Scripting

SFTP latest file from Windows server to Unix host

Hi All, I am SFTP ing file from Windows server (Sydney) to Unix host in London (both servers are on different timezones). It appears the upstream (Windows server team) does not do a file housekeeping activity and henceforth there are multiple files i.e. each week the script is scheduled to run... (1 Reply)
Discussion started by: vigdmab
1 Replies

7. Shell Programming and Scripting

Latest file through SFTP

Hi, I want to download the latest file published on remote server using SFTP connection. The job will run at every 15 min hence should download only the latest file. The fileName will be like ABC_DDHHMISS.txt (DD- day, HH-hour, MI - Min, SS-Sec). So there will be files with same date for... (23 Replies)
Discussion started by: ssachins
23 Replies

8. AIX

Automate SFTP UNIX to Windows

Hi, Could you please help to solve the below issue... my requirement is automate the SFTP between UNIX and Windows server. I want to get and put some files to UNIX AIX machine(SFTP client) to Windows server(SFTP server). For that, i have generated key pair (private/public) in my AIX machine .... (6 Replies)
Discussion started by: mahiban
6 Replies

9. Shell Programming and Scripting

automate sftp using unix script

Hi All, I need to write a UNIX script that automates the sftp process. I should be able to do a sftp to a secure box and get a file from there. I am having a problem doing this because no matter what I do, when I run my script, I get a prompt at command line asking for a password. How could I... (34 Replies)
Discussion started by: priyamurthy2005
34 Replies

10. Shell Programming and Scripting

Automate SFTP is not working

Hi All:cool:, i tried to automate SFTP process after passwordless authendication. Stil i am getting error... Can anyone help.... ------------------- sample code below ------------------- sftp -v $mdskk@100.4.4.75 << EOF cd /data mget *.tar.gz bye EOF... (2 Replies)
Discussion started by: senthil_seera
2 Replies
Login or Register to Ask a Question