Visit Our UNIX and Linux User Community


Use cURL in shell script and get most recent file from remote server using SFTP


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Use cURL in shell script and get most recent file from remote server using SFTP
# 1  
Old 08-20-2009
Use cURL in shell script and get most recent file from remote server using SFTP

I have been trying to implement the following shell script --

sftp to remote server
get list of files in the directory
sftp get the most recent listed file
exit

This script will be scheduled to be executed everyday using CRON or CONTROL-M and the need is to have absolutely no human interaction. This should be executed without having the need to have someone answer system prompts (eg: enter username and password, accept certificate etc.)

I was told that this could be achieved easily using cURL in the sh script. Can someone confirm that this cannot be done without cURL. Can someone help me write this script.
# 2  
Old 08-23-2009
If you are running this script without human interaction, then you need to establish some criteria for what files to get.
Are you always retrieving files from the same directory on the remote server?
Are you always fetching the same number of files? Or all files modified since the last script?

Are you retrieving files from a public server? That is, do you need to supply a username and password, or can anyone retrieve the files.

If it's not a public server, and I assume it's not, do you have secure shell (SSH) configured? I would do that first. If you copy your public key from your PC to the server, you can use ssh, scp and sftp without a password. In a script, scp will be more convenient than sftp.

I would also checkout rsync. It's probably what you really want.
# 3  
Old 08-24-2009
Hi Ken, Thanks for the response. Let me elaborate my situation;

1. Are you always fetching the same number of files? -- always getting one file.
2. criteria for what files to get -- The most recent one i.e. the one with the most recent time stamp. File name will keep changing in an undefinable pattern.
3. Are you retrieving files from a public server? -- no, need to provide username and password.
4. ssh, scp and sftp without a password. In a script, scp will be more convenient than sftp. -- the server limits itself to sftp.

I will check out rsync. Thanks. Based on details above, any suggestions?
# 4  
Old 08-24-2009
Maybe something like this:
Code:
#!/bin/sh
SERVER=example.com
DIR=some_directory
FILE="$(echo ls -t | sftp -b- $SERVER:$DIR | head -n1)"
echo get "$FILE" | sftp -b- $SERVER:$DIR

These individual commands worked for me with no username or password because I have SSH setup to the server. If you don't, I assume you'll have to add user@. I'm not sure how the password is passed.

Command ls -t lists the files in time order, so the most recent file is first. Note that if there are subdirectories that might be modified, one could end up as more recent than any file, which isn't what you want.

Switch -b- tells sftp to work in batch mode and accept commands from stdin, which is the ls command we just piped to it.

Command head -n1 runs on the local machine to catch just the name of the first file.

The syntax FILE="$(...)" runs a command on the local machine and captures the output into a variable named FILE.

Even if the server really doesn't allow ssh, you may still be able to setup SSH keys so that sftp works without a password. It's secure because you can only access it if you log into your local machine.
# 5  
Old 08-27-2009
Thank you very much. I will try this out and let you know how it went!
# 6  
Old 09-10-2009
Ken,

Since I am moving to a dir when I do the sftp, the FILE variable is getting populated with "Changing directory..." message. Can you help me with a workaround., please?

Amit
# 7  
Old 09-10-2009
may be...

Code:
 
FILE="$(echo ls -t | sftp -b- $SERVER:$DIR | head -n2 | tail -n1)"


Previous Thread | Next Thread
Test Your Knowledge in Computers #873
Difficulty: Medium
Self-modifying code is easier to read and maintain because the instructions in the source program listing are always the instructions that will be executed.
True or False?

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to perform checksum of a file on remote server using Curl?

Hi, We have an AIX server and have CURL 7.40 installed in it. We are struggling to perform checksum on the file present on the remote server (AWS). We want to validate the checksum on the file pre and post download to make sure that there is no issue with the file. We are writing a shell... (0 Replies)
Discussion started by: Sanjay_13
0 Replies

2. HP-UX

Shell script to sftp files to file server.

Hi, I am looking for a shell script to sftp to a file server and copy all the files from a directory after the script is run. The server name should be a user input parameter and of-course the username/password as well. Rest all should be handled by the script. I tried with below snippet:-... (2 Replies)
Discussion started by: happysingh
2 Replies

3. Solaris

Script to get files from remote server to local server through sftp without prompting for password

Hi, I am trying to automate the process of fetching files from remote server to local server through sftp. I have the username and password for the remote solaris server. But I need to give password manually everytime i run the script. Can anyone help me in automating the script such that it... (3 Replies)
Discussion started by: ssk250
3 Replies

4. Shell Programming and Scripting

Editing a file on remote server using shell script locally

Hi Scott, My previous post was not for any school or college projects. I am currently working with a IT company (Cannot provide more details than this). I am trying to implement the below script in my day-to-day work, Apologies for the confusion in previous post :). My question remains same... (4 Replies)
Discussion started by: Nishant Ladiwal
4 Replies

5. Shell Programming and Scripting

Editing a file on remote server using shell script locally

Hi All, I have below requirements for my project: 1. Building a shell script which connects to a remote server 2. running script on local machine as user 'A' 3. connecting to server using user 'B' with password 4. login with a powerbroker role 'P' (asks for same password as 'B') on that... (1 Reply)
Discussion started by: Nishant Ladiwal
1 Replies

6. Shell Programming and Scripting

SFTP script to automate login in to remote server

Greetings, guys. I'm not much of a programmer forgive me for being a noob, because of someone leaving, I was put in an IT spot where I have to figure out a few things. Being new to Linux and programming has been a challenge. My boss has asked me to create an automated script to connect to a 3rd... (7 Replies)
Discussion started by: giovannym
7 Replies

7. Shell Programming and Scripting

i want to execute shell script on remote server with in sftp session

Hi, I want to execute shell script with in sftp session for remote server. like i have a shell script test.sh that is on local server.i want to execute that script on remote server sftp user@192.168.56.10 sftp> test.sh ---execute for remote server not for local server. sftp... (3 Replies)
Discussion started by: SAUD PASHA
3 Replies

8. UNIX for Dummies Questions & Answers

Pull a file from a remote server through a shell script

Hi, I am writing a shell script to pull a file from a remote server (Let say its a windows based remote server). One of my criteria is to pull a file only if it is not empty. We have done a similar script to push a file from our end to a remote server and before pushing it we check for the... (2 Replies)
Discussion started by: sashankkrk
2 Replies

9. UNIX for Dummies Questions & Answers

Get the most recent file from a remote server

Hi, This is an FTP related query.I hvae 2 files in a remote server with the timestamp attached to it. FilenameYYYYMMDDHHMMSS.EXT. I need to extract both the fileswith the most recent timestamp using the FTP script I have used : ftp Servername blah blah.... MOST_RECENT_FILE1=`ls -1t... (5 Replies)
Discussion started by: anujairaj
5 Replies

Featured Tech Videos