ftp nightmare


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting ftp nightmare
# 1  
Old 03-17-2010
ftp nightmare

Hi everyone

I have a Fedora FTP server (lets call it FTP-SERVER1) and files are constantly being uploaded to it from a 3rd party.

Once every 15 minutes I need to "move" the files from the FTP
server to another server (lets call it SVR-WINDOWS) where it will be processed and then deleted.

I'm thinking the process should be something like this:
  1. From SVR-WINDOWS, open ftp connection to FTP-SERVER1
  2. list the contents of the "ftp home" directory (e.g. dir = file1, file2, file3)
  3. get each file listed (get file1, get file2, get file3)
  4. delete each file that was copied (delete file1, delete file2, delete file3) to SVR-WINDOWS
  5. close ftp connection

While the above takes place more files are uploaded to the FTP server. So next time I execute my script, it will move the newly uploaded files.

Does anyone know if it's possible to do the above?

If yes, then how do I go about to "list files" and only get and delete the files "listed".

Many thanks

Last edited by rbatte1; 09-26-2016 at 12:38 PM.. Reason: Converted to formatted number-list
# 2  
Old 03-17-2010
After you have uploaded the first batch, make a text file and edit it so that it contains:
Code:
rm file1
rm file2
etc.

At the end of the file add the lines
Code:
mget *
quit

Use this newly created file as input to your next ftp session.
Depending on the size of the files being uploaded from the thrid party. you might retrieve files that are still in the process of being uploaded.
Can you have the third party, send a second file of minimal size indicating the primary upload is complete for that file?

Last edited by rbatte1; 09-26-2016 at 12:38 PM.. Reason: Added CODE tags
# 3  
Old 03-17-2010
hi jgt

i was thinking something like this might work:

Code:
# Get list of files on FTP server
ftp -i -n <<EOF > /tmp/filelist.txt
open <hostname>
user <userid> <passwd>
ls
bye
quit
EOF

# Run through the list of files and get, delete each file
for i in `cat /tmp/filelist.txt`
do
  ftp -i -n <<EOF
  open <hostname>
  user <userid> <passwd>
  get $i
  delete $i
  bye
  quit
  EOF
done

Do you think this will work?

It doesn't solve the problem of copying partially uploaded files (thanks for bringing it up).

---------- Post updated at 02:45 PM ---------- Previous update was at 02:21 PM ----------

Okay tried my script, but it didn't work.

It gets a list of file names, but it's not happy with the second part of the script. It fails at the line with "ftp -i -n <<EOF" (inside the "do" bit).
# 4  
Old 03-17-2010
If there is a problem processing the retrieved file, you have already deleted it from the source.

Code:
#!/bin/ksh
if [ -r lock.file ]
then
 exit 1
fi
echo $$ >lock.file
if [ ! -r input.file ]
then
 echo "mget *" >input.file
 echo "quit " >>input.file
fi
ftp site <input.file
cat /dev/null >input.file
list=`ls received files`
#list=`ls *.job`  if there are two files file.data and file.job 
for file in $list
  process $file
  echo "rm $file\n" >>input.file
  mv $file ../done
done
 echo "mget *\n" >>input.file
 echo "quit\n" >>input file
rm lock.file

Adjust as necessary
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Solaris

AI server corporate nightmare

Hello Ex-Sun fellows, i've been tasked to install a bran spanking new AI Solaris 11.1 server for our Oracle park. With the documentation this is what's running and what's working. T5120 2 oracle VM's (one ldom on each disk) One with a Solaris 11.1 repo, AI server. One for testing... (0 Replies)
Discussion started by: maverick72
0 Replies

2. UNIX for Advanced & Expert Users

Virtualizing SCO v5 - interesting project/nightmare, could use some help

same old story, we have an important very old app running on SCO 5, box is only on LAN and as such was never updated by anyone etc.. box is nearing its end of life in a bad way (currently experiencing some SCSI issues) and it's time to do something when good advice of "lets upgrade" for years was... (1 Reply)
Discussion started by: mc18
1 Replies

3. Solaris

date -d nightmare on Solaris

Hello there ppl, I thought my question would qualify to be posted in this forum and in Shell scripting forum. And I swear to God.. there is no discussion on this exact topic anywhere else on the web! So my script on BASH uses 2 commands: 1) date -d "Fri Mar 06 10:18:16 UTC 2009" +%s ... (1 Reply)
Discussion started by: pavanlimo
1 Replies

4. BSD

FreeBSD nightmare!!!

Dear friends out there, i hope u'll have enough time to read this problem of mine and try to help me solve it. well, i've been a long time user of microsoft products and happened to come across FreeBSD when one fellow referred me to it saying that it was a wonderful OS which one could use for web... (6 Replies)
Discussion started by: kenyatta
6 Replies

5. UNIX for Dummies Questions & Answers

is unix really such a nightmare... or is it me?

i rue the day that my server manager and i parted company... the start of a long journey.... :( sometimes i find myself daydreaming about the days when i could say... "this dont work, can u fix it?".... and 2 mins later it worked! i have a new way of "cursing" at ppls.... i just say "failed... (10 Replies)
Discussion started by: mickeymouse
10 Replies

6. UNIX for Dummies Questions & Answers

installing apache (nightmare for me)

Please help... i'm new to this job and new to unix as well..... i'm trying to install apache 2.2.6 it's installed on one server... i need to install it on another server... my clue was to maybe use the fetch command... please help.....for example..... apache is on 69.50.132.14.... and it needs to... (1 Reply)
Discussion started by: marinob007
1 Replies

7. Shell Programming and Scripting

Splitting Chunked-FullNames Nightmare

I've got a problem i'm hoping other more experienced programmers have had to deal with sometime in their careers and can help me: how to get fullnames that were chunked together into one field in an old database into separate more meaningful fields. I'd like to get the records that nicely fit... (2 Replies)
Discussion started by: RacerX
2 Replies

8. HP-UX

Viewcvs...a nightmare on HP-UX!!!

Hello I'm new on this forum but I have a big problem. I've installed Subversion 1.1.1 and Apache 2.0.52 on a HP-UX. This is the uname: HP-UX xxxx B.11.11 U 9000/800 4169945236 unlimited-user license Now I must to install a software to browse the svn repositories. My choice is Viewcvs. ... (3 Replies)
Discussion started by: goblin79
3 Replies
Login or Register to Ask a Question