Shell Programming and Scripting

Hello,

I have a server that I have to ftp files off and they all start SGRD and are followed by 6 numbers.

Code:

SGRD000001
SGRD000002
SGRD000003

The script I have will run every 10 mins to pick up files as new ones will be coming in all the time and what I want to do is delete the files I have picked up from the ftp site as I don't want to keep downloading the same files.

Once I have downloaded the files I need to rename them. Before the renaming process I have done an ls SGRD* > Myfile on the files I have downloaded and then run a loop to put delete in front of the file names.

Myfile contents

Code:

delete SGRD000001
delete SGRD000002
delete SGRD000003

Is there anyway that I can run the commands within Myfile on the ftp server so that I just delete the files I downloaded ?

Another question I have is, is it possible to move the files that I have downloaded into a backup folder on the ftp server or do I have do download the files and upload them back to the ftp server ?

I tried rename in various different ways and it didn't work with multiple files.

One way to do this.

Code:

# Assume the filenames live in a file called "infile"

echo "USER username password
cd /remote/path/to/files
`for f in cat infile; do
  echo get $f
  echo delete $f
  done`
bye | ftp -n remoteplace

Because you are deleting the files and you cannot go back in time to get them if there is a problem:
I would consider making 2 connections - the first to get the files, check the files, then make a second connection to delete them.

Note: rather than putting passwords in the script consider using a .netrc file, and set the protections to 600. In general, ftp is insecure, and this helps only a little bit.

This whole operation would be much easier if you could do it from the ftp server itself. Do you have sysadmin rights on the ftp server?
Can you 'put' the files from there instead of 'get' the files? You can then move the files already 'put' to another directory.

I don't have any sysadmin rights to the ftp server in question and a team I work with don't want us to have either.

Assuming you have the rights as the account you log in with ftp as, the del subcommand just needs to replace the get subcommand. If you have another way to verify that the transfers completed okay before you issue the deletes, then is there a problem in just looping for each file? There is no doubt a small overhead in putting a single delete command into ftp and then looping for each file, but it would make the code easier:-

Code:

while read file
do
   ftp -inv $host <<-EOFTP >>ftp_log 2>&1
      user $user $password
      cd $target_directory
      del $file
      quit
EOFTP
done < Myfile

The credentials are included in the flow of commands to keep them off the ftp command line, else someone can see them with a simple ps, but like Jim says, it is better to use entries in a .netrc file. If you do, drop the -n flag or it will ignore the .netrc file. better still, if you can convert to sftp using ssh-keys to automate authentication it becomes easier again, but that may not be possible in your environment. It depends what you have for the client and the server and you need access on port 22 instead, so that might be one or more firewalls with a rule change.

If you just need to get the files and delete them, I would suggest two ftp commands per file so that you can check the transfer completes before you delete the source, something like:-

Code:

while read file
   do
      ftp -inv $host <<-EOFTP >>ftp_log.$file 2>&1
      user $user $password
      cd $target_directory
      get $file
      quit
EOFTP

# Verify here that the file arrived okay by reading the log ftp_log.$file
# If okay, then proceed with delete step

   if [ whatever test you decide ]
   then
      ftp -inv $host <<-EOFTP >>ftp_log.$file 2>&1
         user $user $password
         cd $target_directory
         del $file
         quit
EOFTP

   else
      # Whatever error routine you decide
   fi
done < Myfile

Robin

The delete loop works a treat but what I have found is that if I download a lot of small files then I get the following message

Code:

425 Unable to build data connection: Cannot assign requested address
local: SGRD097740.sav remote: SGRD097740.sav

and the files are left on the ftp site.

Is there a sleep or pause command in ftp ? or would you just rerun the ftp job again to pick up any outstanding ones that have the above message ?

This is probably the server being unable to respond quickly enough. Just putting a small sleep in the loop outside the actual ftp should help. Of course if you sleep for 1 second but have 600 files that will take 20 minutes to complete if you sleep between every ftp and you open a second connection to do the deletion.

Does there seem to be a common limit to the number of connections when getting lots of small files? Perhaps you can have a sleep every 10 files so it doesn't hurt too much.




Robin