How to check if downloading of a file is completed?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to check if downloading of a file is completed?
# 1  
Old 01-07-2017
How to check if downloading of a file is completed?

Hello All.
We are getting one zip file from another server daily. The size of the zip file will be around 4.5 gb that takes time to download completely. I have to process that file furthest once get downloaded completely. I have written one code for that, but i need suggestion whether it is a right approach or any better way with no surprises. Kindly suggest
Thanks in Advance

I am comparing size of file after every 60 seconds.

Code:
#creating function
_func() {

x=`du -sk "$path" | awk '{ print $1 }'`
sleep 5
y=2

while [ "$x" != "$y" ]
do
y=`du -sk "$path" | awk '{ print $1 }'`
echo "size before sleep is $y"
echo "sleep for another 60 seconds"
sleep 60
x=`du -sk "$path" | awk '{ print $1 }'`
echo "size after sleep $x"
done

}

#using function
if _func ; then
echo "downloading is complete"
else
echo "downloading"
fi

# 2  
Old 01-07-2017
You can use "stat" to get the exact size of the file.("du" is quite the same here).

If you want to be sure the file is correctly downloaded, create a checksum at the source and verify the checksum after downloading. Use "md5sum" or the "sha1sum" command-family for this.
# 3  
Old 01-07-2017
In addition to what stomp already said... Download the checksum in a separate file AFTER the download of the main file completes on the server that is downloading the files. Then just look for the 2nd file containing the checksum on the receiving client instead of monitoring the download of the 1st file.
# 4  
Old 01-08-2017
Quote:
Originally Posted by Don Cragun
In addition to what stomp already said... Download the checksum in a separate file AFTER the download of the main file completes on the server that is downloading the files. Then just look for the 2nd file containing the checksum on the receiving client instead of monitoring the download of the 1st file.
Thanks stomp
Hello Mr. Don, the third party is sending us a zip file , presently they are not sending checksum but it is a good idea to ask them also to send checksum. But i don't understand by your statement
PHP Code:
Download the checksum in a separate file AFTER the download of the main file 
. I guess i need to compare checksum of the file at our end with the checksum of what the other party would send ? . In that case also i need to watch continuously if the checksum have matched or not by using the same while loop. I also don't understand why not to monitoring the file.
# 5  
Old 01-08-2017
Let us propose that your 3rd party send you a zip file named file.zip (as they are doing now) and after they have sent you file.zip they then send you a file named file.checksum containing the checksum for file.zip. I am suggesting that instead of trying to figure out when file.zip is complete by trying to determine when it has stopped growing, you instead just watch for non-zero length files with names ending in .checksum. For each non-zero length file.checksum you find you will know that you have a complete corresponding file.zip and you can verify that it was received correctly by comparing the contents of file.checksum to the checksum of the file file.zip.
# 6  
Old 01-08-2017
Quote:
Originally Posted by Don Cragun
Let us propose that your 3rd party send you a zip file named file.zip (as they are doing now) and after they have sent you file.zip they then send you a file named file.checksum containing the checksum for file.zip. I am suggesting that instead of trying to figure out when file.zip is complete by trying to determine when it has stopped growing, you instead just watch for non-zero length files with names ending in .checksum. For each non-zero length file.checksum you find you will know that you have a complete corresponding file.zip and you can verify that it was received correctly by comparing the contents of file.checksum to the checksum of the file file.zip.
Thanks Mr Don, this seems very valid approach to wait for completion of transfer of main file by watching it's checksum file that will arrive at end. But just for a knowledge how third party will handle this. Presently they are manually uploading file to our end, So they can also send the checksum file for now but how this can be automated. How code can recognise if the uploading is completed and checksum file can be transferred.
Thanks.
# 7  
Old 01-08-2017
What commands are the 3rd party currently using to transfer a "main" file?

What operating system is being used on the 3rd party's server from which the "main" file is being sent?

What operating system is being used on your client machine to which the "main" file is being sent?
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

File downloading

I saved to disk using lynx, under ls it shows up but no matter what I have done back comes the statment no such file. How do you erase these ls false readings or open what ls shows? (9 Replies)
Discussion started by: Jackbird
9 Replies

2. Shell Programming and Scripting

Downloading file from mainframe to UNIX

Hi All , I need a help regarding file ftp ing from mainframe to unix.Our source file is mainframe file.I need to download the mainframe file to local unix server through unix script.If anyone can help me how we can do it through unix script ,it will be really helpful.Thanks. (7 Replies)
Discussion started by: STCET22
7 Replies

3. UNIX for Dummies Questions & Answers

Need help in not fetching a file while file writing operation is not completed

Hi All, We have a Unix program in oracle when we run the program this connects to specified ftp and will get the file into local server. We are facing a problem like when file writing operations is not completed, this program is getting the incomplete file. Could anyone please help me... (2 Replies)
Discussion started by: world.apps
2 Replies

4. Shell Programming and Scripting

File downloading help in perl

Im trying to write a perl code to do the following task. I have a website which is located in a diffirent country. I want to download a file from that site everyday. The file name keeps on changing everyday. say the file name is CTP-latest-5.3.0.37.iso next day it may change to... (0 Replies)
Discussion started by: srijith
0 Replies

5. UNIX for Dummies Questions & Answers

Unix command used for downloading file from net

Hi, Which command I can use to download a file from website? I tried with wget and lwp-dowload but no gain. Can any one suggest me the good way to dowload? (2 Replies)
Discussion started by: siba.s.nayak
2 Replies

6. Shell Programming and Scripting

File downloading

Hi, I am newbie in Perl/CGI and I need your help to develop a little script to help me to download some files located in a FTP server via HTTP to a disk. I need to authenticate with username/password via a HTTP page and I can choose the files to download from the web page to my disk. Can... (2 Replies)
Discussion started by: imad77
2 Replies

7. Shell Programming and Scripting

Downloading file from internet

Hi, I need to write a script to download a file (say http: //www.mysite.com/myfile.txt) from the internet (please note that the file is on http not ftp). Please let me know if there is any method other than wget. Thanks, Chetan (4 Replies)
Discussion started by: chetanpc
4 Replies

8. Shell Programming and Scripting

Downloading a csv file automatically

I want to automatically download a CSV file daily which can be found here: London Stock Exchange - Search and the link is named "Click to download covered warrants (100Kb)" on the right hand side. What commands can I use to invoke clicking on this link and initiating the download? And also... (2 Replies)
Discussion started by: figaro
2 Replies

9. UNIX for Dummies Questions & Answers

Downloading Excel File

Hello All, Currently working with CGI scripts in perl which involves a) uploading b) processing c) downloading the file back to the user uploading and processing quite trivial and they are done Third part is kind of being tricky. Excel file in binary format is getting displayed in the... (4 Replies)
Discussion started by: matrixmadhan
4 Replies
Login or Register to Ask a Question