Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

How to check if downloading of a file is completed?

Shell Programming and Scripting


Tags
bash

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 01-07-2017
looney looney is offline
Registered User
 
Join Date: Oct 2015
Last Activity: 21 February 2017, 9:38 AM EST
Posts: 84
Thanks: 60
Thanked 11 Times in 11 Posts
How to check if downloading of a file is completed?

Hello All.
We are getting one zip file from another server daily. The size of the zip file will be around 4.5 gb that takes time to download completely. I have to process that file furthest once get downloaded completely. I have written one code for that, but i need suggestion whether it is a right approach or any better way with no surprises. Kindly suggest
Thanks in Advance

I am comparing size of file after every 60 seconds.


Code:
#creating function
_func() {

x=`du -sk "$path" | awk '{ print $1 }'`
sleep 5
y=2

while [ "$x" != "$y" ]
do
y=`du -sk "$path" | awk '{ print $1 }'`
echo "size before sleep is $y"
echo "sleep for another 60 seconds"
sleep 60
x=`du -sk "$path" | awk '{ print $1 }'`
echo "size after sleep $x"
done

}

#using function
if _func ; then
echo "downloading is complete"
else
echo "downloading"
fi

Sponsored Links
    #2  
Old Unix and Linux 01-07-2017
stomp stomp is online now
Registered User
 
Join Date: May 2016
Last Activity: 27 February 2017, 6:22 AM EST
Posts: 178
Thanks: 9
Thanked 63 Times in 58 Posts
You can use "stat" to get the exact size of the file.("du" is quite the same here).

If you want to be sure the file is correctly downloaded, create a checksum at the source and verify the checksum after downloading. Use "md5sum" or the "sha1sum" command-family for this.
Sponsored Links
    #3  
Old Unix and Linux 01-07-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is online now Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 27 February 2017, 6:25 AM EST
Location: San Jose, CA, USA
Posts: 10,031
Thanks: 491
Thanked 3,469 Times in 2,952 Posts
In addition to what stomp already said... Download the checksum in a separate file AFTER the download of the main file completes on the server that is downloading the files. Then just look for the 2nd file containing the checksum on the receiving client instead of monitoring the download of the 1st file.
    #4  
Old Unix and Linux 01-08-2017
looney looney is offline
Registered User
 
Join Date: Oct 2015
Last Activity: 21 February 2017, 9:38 AM EST
Posts: 84
Thanks: 60
Thanked 11 Times in 11 Posts
Quote:
Originally Posted by Don Cragun View Post
In addition to what stomp already said... Download the checksum in a separate file AFTER the download of the main file completes on the server that is downloading the files. Then just look for the 2nd file containing the checksum on the receiving client instead of monitoring the download of the 1st file.
Thanks stomp
Hello Mr. Don, the third party is sending us a zip file , presently they are not sending checksum but it is a good idea to ask them also to send checksum. But i don't understand by your statement
PHP Code:
Download the checksum in a separate file AFTER the download of the main file 
. I guess i need to compare checksum of the file at our end with the checksum of what the other party would send ? . In that case also i need to watch continuously if the checksum have matched or not by using the same while loop. I also don't understand why not to monitoring the file.
Sponsored Links
    #5  
Old Unix and Linux 01-08-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is online now Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 27 February 2017, 6:25 AM EST
Location: San Jose, CA, USA
Posts: 10,031
Thanks: 491
Thanked 3,469 Times in 2,952 Posts
Let us propose that your 3rd party send you a zip file named file.zip (as they are doing now) and after they have sent you file.zip they then send you a file named file.checksum containing the checksum for file.zip. I am suggesting that instead of trying to figure out when file.zip is complete by trying to determine when it has stopped growing, you instead just watch for non-zero length files with names ending in .checksum. For each non-zero length file.checksum you find you will know that you have a complete corresponding file.zip and you can verify that it was received correctly by comparing the contents of file.checksum to the checksum of the file file.zip.
Sponsored Links
    #6  
Old Unix and Linux 01-08-2017
looney looney is offline
Registered User
 
Join Date: Oct 2015
Last Activity: 21 February 2017, 9:38 AM EST
Posts: 84
Thanks: 60
Thanked 11 Times in 11 Posts
Quote:
Originally Posted by Don Cragun View Post
Let us propose that your 3rd party send you a zip file named file.zip (as they are doing now) and after they have sent you file.zip they then send you a file named file.checksum containing the checksum for file.zip. I am suggesting that instead of trying to figure out when file.zip is complete by trying to determine when it has stopped growing, you instead just watch for non-zero length files with names ending in .checksum. For each non-zero length file.checksum you find you will know that you have a complete corresponding file.zip and you can verify that it was received correctly by comparing the contents of file.checksum to the checksum of the file file.zip.
Thanks Mr Don, this seems very valid approach to wait for completion of transfer of main file by watching it's checksum file that will arrive at end. But just for a knowledge how third party will handle this. Presently they are manually uploading file to our end, So they can also send the checksum file for now but how this can be automated. How code can recognise if the uploading is completed and checksum file can be transferred.
Thanks.
Sponsored Links
    #7  
Old Unix and Linux 01-08-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is online now Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 27 February 2017, 6:25 AM EST
Location: San Jose, CA, USA
Posts: 10,031
Thanks: 491
Thanked 3,469 Times in 2,952 Posts
What commands are the 3rd party currently using to transfer a "main" file?

What operating system is being used on the 3rd party's server from which the "main" file is being sent?

What operating system is being used on your client machine to which the "main" file is being sent?
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Need help in not fetching a file while file writing operation is not completed world.apps UNIX for Dummies Questions & Answers 2 04-18-2012 01:29 AM
File downloading help in perl srijith Shell Programming and Scripting 0 07-21-2011 08:08 AM
File downloading imad77 Shell Programming and Scripting 2 10-22-2008 12:38 AM
Downloading file from internet chetanpc Shell Programming and Scripting 4 03-25-2008 05:36 PM
Downloading Excel File matrixmadhan UNIX for Dummies Questions & Answers 4 01-16-2008 01:14 PM



All times are GMT -4. The time now is 07:48 AM.