Download files every one second using ftp script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Download files every one second using ftp script
# 1  
Old 05-27-2013
Question Download files every one second using ftp script

Our main Server "Srv1" is used to generate text files based on specified criteria and it is also connected to two clients (pc1 and pc2) which are responsible for getting the files from Srv1 as it follows:
1. pc1 ( which represents my UNIX machine ) uses shell script to copy the files from Srv1
2. pc2 ( which is represent my colleague window machine ) uses another technique to cut the file from Srv1

my shell script includes below ftp session to copy the files from Srv1 to pc1 , In fact I run it for 2 days as cronjob for every minute and it was working fine but while did compare the collected files on pc1 with collected files on pc2 I found that pc2 has files more than what pc1 has ( cause pc2 cut some files faster than pc1 which gets the files every one minute ) !!!

SmilieI want to run my script every 1 second or less than if possible to copy the files to pc1 before going to be taken or cut by pc2 , any clue ???

Code:
ftp -in x,x,x,x  << ENDFTP
user username password
cd $ddir
lcd $sdir
bin
prompt
mget *.text
bye
ENDFTP

# 2  
Old 05-27-2013
Before allowing 'pc2' to get/remove the file(s) it needs to make sure that 'pc1' has gotten the file(s), there are multiple ways to accomplish this, one way is to have 'pc1' create a 0 byte file in the directory on the host where it got the file(s) for each file(s) it successfully got, then the script on 'pc2' would only get/remove the file(s) that have been successfully processed by 'pc1' and when 'pc2' got a file it would also remove the temp file that told it it could get remove the file.
# 3  
Old 05-27-2013
Does Svr1 support inotify(is it Linux?)

You should NOT poll for files as you are doing things. Svr1 knows when a local file is closed, it is hard to do that reliably using ftp from a remote connection.

Set up a file watch on Svr1, have it move files to a special subdirectory. Any file in that subdirectory is complete and ready to ftp. Run ftp once per minute - have it list files in that special subdirectory. Use the output of that as an input list of files to grab.

Running ftp every second is a disaster waiting to happen. Consider something else.
# 4  
Old 05-27-2013
I have no right to play with Srv1 ..

---------- Post updated at 04:42 PM ---------- Previous update was at 04:41 PM ----------

Quote:
Originally Posted by spacebar
Before allowing 'pc2' to get/remove the file(s) it needs to make sure that 'pc1' has gotten the file(s), there are multiple ways to accomplish this, one way is to have 'pc1' create a 0 byte file in the directory on the host where it got the file(s) for each file(s) it successfully got, then the script on 'pc2' would only get/remove the file(s) that have been successfully processed by 'pc1' and when 'pc2' got a file it would also remove the temp file that told it it could get remove the file.

please do the code for me ..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Ubuntu

Bash script for FTP download -Mysql

Hi guys, I recently managed to write up my working script, but now I have a problem. If the file isn't there in the remote server, my actual script jumps it and all ok, but I need something like this: Search file -> if there, then download -> if not, download next file in the list. Any... (7 Replies)
Discussion started by: virtus96
7 Replies

2. UNIX for Dummies Questions & Answers

How to download files matching pattern from FTP using CURL or WGET?

Hi, For an order I requested, the provider has uploaded a tar file in public FTP site which internally has tons of files (compressed) and I need to download files that follows particular pattern which would be few hundreds. Note: The order can't be requested for files that follows the... (7 Replies)
Discussion started by: Amalan
7 Replies

3. Shell Programming and Scripting

Curl ftp ssl download files

Hello all, I have been struggling with this issue on and off for a couple of weeks now and I just got it all working, so I wanted to share my findings in case some other poor soul needs to know how. First some background on what I'm doing. I am uploading files to different directories based on... (0 Replies)
Discussion started by: msjkadams
0 Replies

4. Shell Programming and Scripting

FTP download using perl script

Need assistance I have a script which i can download the files from ftp server using perl . But i want to download multiple files at a time #!/usr/bin/perl -w use Net::FTP; $ftp = Net::FTP->new("ftp.ncdc.noaa.gov"); $ftp->login('username', 'password'); $ftp->cwd("<dir>");... (9 Replies)
Discussion started by: ajayram_arya
9 Replies

5. Shell Programming and Scripting

Need of shell script to download data using ftp

Hi ! I am just wondering shell script to download data from ftp... I have text file containing ftp address...looks like this ftp://site...../filename.xyz ftp://site...../filename.xyz ftp://site...../filename.xyz ftp://site...../filename.xyz script has to read ftp address and... (8 Replies)
Discussion started by: nex_asp
8 Replies

6. Shell Programming and Scripting

Help on FTP download using UNIX script

Hi guys, I'm new on this forum and on UNIX. Can somebody help in writing a script to download a file from an FTP server and validating if there is a file to download. If there is a file, I would send it to a mail recipient and if not I would generate an error log. Thanks in advance!:D (1 Reply)
Discussion started by: rjay_45
1 Replies

7. Shell Programming and Scripting

ftp - How to download files which created today?

Hello experts, I have written following script to download files which created today. Unfortunately, it's not working. test.ksh: #Defining variables USR='xxx' PASSWD='yyyy' HT='test.test.com' FILE='S*.pdf' XFILE=$(echo find . -type f -mtime 0) ZFILE=$(echo ls -tR|grep 'Jun 8')... (14 Replies)
Discussion started by: dipeshvshah
14 Replies

8. Shell Programming and Scripting

Automatic FTP-Download | not files older then x days

Hey Guys, i need to download files from a ftp-server that not older than $VAR (x) days eg for seven days ./script 7 or two weeks ./script 14 all files from the last 14 days will download but how i can explain "ftp" this? sorry for my strange english :/ (2 Replies)
Discussion started by: tetex
2 Replies

9. Shell Programming and Scripting

Download previous day files through FTP

Hi All, I have a scenario where I need to download the previous day modified files from other server to my server through FTP . Could any one please send me the shell script for the same. I used the following but I dont know how to proceed after this. ftp -n -i -v $IP <<ENDOFinPUT >>... (3 Replies)
Discussion started by: sarathchandrach
3 Replies

10. Shell Programming and Scripting

script for download files from ftp site

I'm new to scripting. I'm trying to write a script to download files from ftp site, the following is the script and the message i get after running the script. no files were downloaded :( Thanks advance! script: #!/usr/bin/ksh DAY=`date --date="-1 days" +%y%m%d` ftp -v -n "ftp.address" <<... (5 Replies)
Discussion started by: tiff-matt
5 Replies
Login or Register to Ask a Question