lftp script to connect to external sftp site and download to internal ftp and then send email


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting lftp script to connect to external sftp site and download to internal ftp and then send email
# 1  
Old 10-17-2012
Computer lftp script to connect to external sftp site and download to internal ftp and then send email

Hi there, I'm new to shell scripting and need some help if possible?

I need to create a shell script (.sh) to run as a cron job on an ubuntu linux server to connect to an external sftp sites directory using credentials (which I have) and then download to our internal ftp server and then copy off to another share which is an archive location. I would like to remove the files from the external sftp site after the data has been downloaded successfully.
After the above is complete we need the script to send a notification email showing the list of file names that have been downloaded.

I have been looking at lftp as the client for this process and have put together the following mainly just the parameters, not alot I know but it's a start and I'm still learning as I go Smilie

Code:
#!/bin/bash
# FTP settings
HOST='ftpdownload.co.uk'
USER='user'
PASSWD='password' 
MNTDIREVF='download directory'
MNTARCEVF='Archive of download directory'
# Email settings
SUBJECT=" FTP Download"
EMAIL="joe.bloggs@email.co.uk"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`
 
# Start of main script
# Connect to remote server using lftp and sync 'down' 
lftp -u user,password sftp://website <<EOD
cd to_gbo
cd DMEVF
mirror --Remove-source-files /to_gbo/DMEVF
?
?

If anybody has experience in creating such a script and can offer any advice I would be ever so grateful.

Last edited by Scott; 10-17-2012 at 04:31 PM.. Reason: Please use code tags
# 2  
Old 10-17-2012
How many hosts are we talking about? Sftp from A /somewhere to local B /somewhere2 then ftp from local B /somewhere2 to ftp C /somewhere3, and cp to share locally mounted /somewhere4.

Also, you need to verify you got a good copy, as you are destroying the original. You can get a length from ls. You can dlowload twice. Better if you signall they can remove it later by moving/renaming it somewhere else on that server (preferably on same device, so not a copying operation).

sftp reads /dev/tty for passwords, so PPK would be a lot easier than sftp under 'expect' or 'ssh -tt' or something to capture a fake tty you can write a password to, plus it is bad security to have passwords kicking around.

Making a file list with ls under sftp and keeping it to drive email is pretty simple.
# 3  
Old 10-18-2012
Thanks for the post.

We are talking about two hosts on our network, one of these is our internal linux FTP server (which manages all of our FTP downloads) and the other host is our windows fileshare server, this is where the data needs to be transferred to. After the data is transferred, an archive copy needs to be copied to another folder on the fileshare server with the folder being the "date" of when created.

We can only download the data once from the external ftp server due to the amount of data. This transfer will take place daily. We currently use lftp so I would like to continue using this command if possible....

I will continue trawling the web to get a better understanding, I'm just unsure of the code and how to arrange it in the script as it confuses the hell out of me.
Smilie
# 4  
Old 10-18-2012
I guess the security model is important. Is the FTP serer on the open Internet and the archive behind a firewalL, that mounting one onto the other is a problem? With a mount, it is just mkdir and mv. If you need to ftp the data out of the FTP server, sometimes it is simpler to push, as the files to be discovered as new and not being written are local, but more secure to have the archive server pull, so there is no ingoing login credentials or trust. Even with pull, the FTP server can 'stage' the data to be archived in a dedicated sub-tree so the archive server just gets anything in there to a parallel path.

Renaming and moving files inside a device is a great way to give them a status indicator, so they are not moved prematurely. Unlike a cp, where files take time to write, with mv the files are not there one moment, and there the next, complete. It's just a directory write. Ack or marker files is just muddy, old guy thinking.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Lftp sftp get - script renames the local file with suffix tilde

Hi, Below script used for sftp get, #/bin/bash USER=xxx PASS=xxx HOST=xxx REMOTE_FILE=$1 LOCAL_FILE_LOC=$2 cd $LOCAL_FILE_LOC lftp sftp://$USER:$PASS@$HOST:10022 -e "get $REMOTE_FILE; bye" If file does not exist in sftp server, and file (same as remote file name) exists in local dir,... (4 Replies)
Discussion started by: vhegde1011
4 Replies

2. Cybersecurity

When i start CSF i cant connect VPS or download any data into it It appears i cant connect Linux VP?

It appears i cant connect linux VPS server via SSH or i cant SCP any file to it and i cant wget any file TO it (from inside it) while CSF (Config Server Firewall, LFD is running. Just after isntall in default configuration and after changing TESTING mode to LIVE mode. Trying to wget & install... (1 Reply)
Discussion started by: postcd
1 Replies

3. Red Hat

Disk in Linux machine are internal or external

How to check whether the disk in linux machine are internal or external ( from nas or san). How to identify internal(local) and external disks. Following are some details of my server. Thanks. #df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 15G 3.5G 10G... (2 Replies)
Discussion started by: salmanraza
2 Replies

4. Shell Programming and Scripting

Script to send email after comparing the folder permissions to a certain permission & send email

Hello , I am trying to write a unix shell script to compare folder permission to say drwxr-x-wx and then send an email to my id in case the folders don't have the drwxr-x-wx permissions set for them . I have been trying to come up with a script for few days now , pls help me:( (2 Replies)
Discussion started by: nairshar
2 Replies

5. UNIX for Advanced & Expert Users

lftp: Option to lftp a file, wait and download a file as soon as its created

Please let me know what is lftp options combination to wait and download a file from target as soon as its gets created. I tried with different options but not able to get it working as I need any help would be appreciated (4 Replies)
Discussion started by: bmkux
4 Replies

6. Shell Programming and Scripting

LFTP - to download files that are between 0 and 3 days old

Im writing a script and i specifically need it to download files that are between 0-2 days old. This will run every 2 days. I understand lftp supports newer files only, but these files will be removed from the target so this is not what we want. Does anyone know how to do this? ----------... (0 Replies)
Discussion started by: mokachoka
0 Replies

7. Linux

shell script to download files from a site?

Hey everyone, my wife has purchased a bundle package of a bunch of images from a site, and now has to download each one of them manually. There are about 500 downloads, and it's quite a hassle to browse to each page and download them all individually. I would like to write a shell script to... (2 Replies)
Discussion started by: paqman
2 Replies

8. Shell Programming and Scripting

script for download files from ftp site

I'm new to scripting. I'm trying to write a script to download files from ftp site, the following is the script and the message i get after running the script. no files were downloaded :( Thanks advance! script: #!/usr/bin/ksh DAY=`date --date="-1 days" +%y%m%d` ftp -v -n "ftp.address" <<... (5 Replies)
Discussion started by: tiff-matt
5 Replies

9. Shell Programming and Scripting

need script to connect sftp servers

Dear friends, i need to connect sftp server from my home directory using script . Please can anyone help me on this. (1 Reply)
Discussion started by: kittusri9
1 Replies
Login or Register to Ask a Question

Featured Tech Videos