Shell Programming and Scripting

Hi there, I'm new to shell scripting and need some help if possible?

I need to create a shell script (.sh) to run as a cron job on an ubuntu linux server to connect to an external sftp sites directory using credentials (which I have) and then download to our internal ftp server and then copy off to another share which is an archive location. I would like to remove the files from the external sftp site after the data has been downloaded successfully.
After the above is complete we need the script to send a notification email showing the list of file names that have been downloaded.

I have been looking at lftp as the client for this process and have put together the following mainly just the parameters, not alot I know but it's a start and I'm still learning as I go

Code:

#!/bin/bash
# FTP settings
HOST='ftpdownload.co.uk'
USER='user'
PASSWD='password' 
MNTDIREVF='download directory'
MNTARCEVF='Archive of download directory'
# Email settings
SUBJECT=" FTP Download"
EMAIL="joe.bloggs@email.co.uk"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`
 
# Start of main script
# Connect to remote server using lftp and sync 'down' 
lftp -u user,password sftp://website <<EOD
cd to_gbo
cd DMEVF
mirror --Remove-source-files /to_gbo/DMEVF
?
?

If anybody has experience in creating such a script and can offer any advice I would be ever so grateful.

How many hosts are we talking about? Sftp from A /somewhere to local B /somewhere2 then ftp from local B /somewhere2 to ftp C /somewhere3, and cp to share locally mounted /somewhere4.

Also, you need to verify you got a good copy, as you are destroying the original. You can get a length from ls. You can dlowload twice. Better if you signall they can remove it later by moving/renaming it somewhere else on that server (preferably on same device, so not a copying operation).

sftp reads /dev/tty for passwords, so PPK would be a lot easier than sftp under 'expect' or 'ssh -tt' or something to capture a fake tty you can write a password to, plus it is bad security to have passwords kicking around.

Making a file list with ls under sftp and keeping it to drive email is pretty simple.

Thanks for the post.

We are talking about two hosts on our network, one of these is our internal linux FTP server (which manages all of our FTP downloads) and the other host is our windows fileshare server, this is where the data needs to be transferred to. After the data is transferred, an archive copy needs to be copied to another folder on the fileshare server with the folder being the "date" of when created.

We can only download the data once from the external ftp server due to the amount of data. This transfer will take place daily. We currently use lftp so I would like to continue using this command if possible....

I will continue trawling the web to get a better understanding, I'm just unsure of the code and how to arrange it in the script as it confuses the hell out of me.

I guess the security model is important. Is the FTP serer on the open Internet and the archive behind a firewalL, that mounting one onto the other is a problem? With a mount, it is just mkdir and mv. If you need to ftp the data out of the FTP server, sometimes it is simpler to push, as the files to be discovered as new and not being written are local, but more secure to have the archive server pull, so there is no ingoing login credentials or trust. Even with pull, the FTP server can 'stage' the data to be archived in a dedicated sub-tree so the archive server just gets anything in there to a parallel path.

Renaming and moving files inside a device is a great way to give them a status indicator, so they are not moved prematurely. Unlike a cp, where files take time to write, with mv the files are not there one moment, and there the next, complete. It's just a directory write. Ack or marker files is just muddy, old guy thinking.