Deleting multiple files off an ftp server once they have been downloaded
Hello,
I have a server that I have to ftp files off and they all start SGRD and are followed by 6 numbers.
The script I have will run every 10 mins to pick up files as new ones will be coming in all the time and what I want to do is delete the files I have picked up from the ftp site as I don't want to keep downloading the same files.
Once I have downloaded the files I need to rename them. Before the renaming process I have done an ls SGRD* > Myfile on the files I have downloaded and then run a loop to put delete in front of the file names.
Myfile contents
Is there anyway that I can run the commands within Myfile on the ftp server so that I just delete the files I downloaded ?
Another question I have is, is it possible to move the files that I have downloaded into a backup folder on the ftp server or do I have do download the files and upload them back to the ftp server ?
I tried rename in various different ways and it didn't work with multiple files.
Last edited by rbatte1; 02-25-2015 at 10:30 AM..
Reason: Added ICODE tags
One way to do this.
Because you are deleting the files and you cannot go back in time to get them if there is a problem:
I would consider making 2 connections - the first to get the files, check the files, then make a second connection to delete them.
Note: rather than putting passwords in the script consider using a .netrc file, and set the protections to 600. In general, ftp is insecure, and this helps only a little bit.
This User Gave Thanks to jim mcnamara For This Post:
This whole operation would be much easier if you could do it from the ftp server itself. Do you have sysadmin rights on the ftp server?
Can you 'put' the files from there instead of 'get' the files? You can then move the files already 'put' to another directory.
Assuming you have the rights as the account you log in with ftp as, the del subcommand just needs to replace the get subcommand. If you have another way to verify that the transfers completed okay before you issue the deletes, then is there a problem in just looping for each file? There is no doubt a small overhead in putting a single delete command into ftp and then looping for each file, but it would make the code easier:-
The credentials are included in the flow of commands to keep them off the ftp command line, else someone can see them with a simple ps, but like Jim says, it is better to use entries in a .netrc file. If you do, drop the -n flag or it will ignore the .netrc file. better still, if you can convert to sftp using ssh-keys to automate authentication it becomes easier again, but that may not be possible in your environment. It depends what you have for the client and the server and you need access on port 22 instead, so that might be one or more firewalls with a rule change.
If you just need to get the files and delete them, I would suggest two ftp commands per file so that you can check the transfer completes before you delete the source, something like:-
Note that the EOFTP to close the input to the ftp command must be either in the first column or tab indented. Space indented will cause it to be ignored and invalidate the input.
The delete loop works a treat but what I have found is that if I download a lot of small files then I get the following message
and the files are left on the ftp site.
Is there a sleep or pause command in ftp ? or would you just rerun the ftp job again to pick up any outstanding ones that have the above message ?
Last edited by rbatte1; 02-26-2015 at 08:13 AM..
Reason: Added CODE tags
This is probably the server being unable to respond quickly enough. Just putting a small sleep in the loop outside the actual ftp should help. Of course if you sleep for 1 second but have 600 files that will take 20 minutes to complete if you sleep between every ftp and you open a second connection to do the deletion.
Does there seem to be a common limit to the number of connections when getting lots of small files? Perhaps you can have a sleep every 10 files so it doesn't hurt too much.
Hi,
I want to delete a file on the local server, while connected to remote server through FTP.
I am using the below code for this
$FTP_CMD -v -n $HOST <<*! >> $LOGFILE 2>&1
user $USER $PASSWORD
cd $DIR
... (11 Replies)
Hi Friends,
I have a requirement to get multiple files from ftp(remote) server and once the files is copied to local machine , I need to move the files on to a different directory in ftp machine.
FTP Machine : 9.9.999.999
Source File Directory : /ftpuser
File Pattern: TMS*
Now I have... (1 Reply)
Hi,
I'm new to shell script..I have one requriement like -
In one server have more than one files,I want to ftp those files to some otehr server..
Ex : test1.pdf
test2.pdf
Please suggest me how to do (3 Replies)
Iam putting file in ftp server.
iam doing ftp
to transfer a file to ftp server
but after sometime(10 sec) the file is automatically deleting in the ftp.
Can i know why this happens.
When my friend ftp the file to the same server , the file is not deleting aftersometime... it is there.
Can... (1 Reply)
Hi,
I'm quite new to unix and perl scripting and need to write a script to delete files older than 7 days from a remote ftp server. Unix or Perl script would do...
I wrote the following code:
#!/usr/local/bin/perl
use File::Basename;
use Net::FTP;
#use Net::FTP::File;
my $verbose =... (15 Replies)
Hi All,
I'm developing a FTP script as below:
ftp -v -n <IP_ADDRESS> << EOF
user avery jSqaqUU2
lcd /directory/folder/
ascii
prompt
mget *
bye
EOF
I would like to enhance the script to count the total file downloaded. For example, once the script run i want the message "Total <n>... (1 Reply)
Hi,
I am facing a weired problem in my FTP script. I want to transfer multiple files from remote server to local server everyday, using mget * in my script. I also, want to send an email for successful or failed FTP. My script works for file transfer, but it don't send any mail. There is... (2 Replies)
I need to download some files from a remote server using ftp. I have ftp'd into the site. I then do an mget * to retrieve all of the data files. Everything seems to proceed normally and I am given feedback that the files were downloaded. Now if I go into the DOS Shell or Windows explorer, it list... (5 Replies)
Hi,
I have a situation where I need to delete multiple files from a folder once I connect to FTP server. I am using ftp script to get the files, number of files always vary from 1 to 100. once I get the files I need to delete all the files downloaded I am making a list of all the files... (4 Replies)