How to validate files after FTP


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to validate files after FTP
# 1  
Old 08-31-2010
How to validate files after FTP

Hi,

I have to write a script and the requirment where using ftp i need to get 8 files from a remote box to local. I know how how to ftp and do get. However i want do a validation after get.

After geting for 8 files, i want to compare the size of the 8 files when it was in remote box with size they have in the local box and depending on that i will dicide if the transfer is suceesfull. Now, i have no i dea how to do this validation.

could anyone please help on this.
# 2  
Old 08-31-2010
checking size is not enough. use a check sum utility like cksum or md5sum and compare hashes
# 3  
Old 08-31-2010
With anything to do with ftp it helps to know which Operating System(s) are involved and whether you have administrative access to both computers to write scripts etc..

Also:
Are these Binary or Text files?
Which end is initiating the ftp and whether it is a "get" or a "put"?
This User Gave Thanks to methyl For This Post:
# 4  
Old 08-31-2010
you can do something like this:---
Code:
for old files 
for file in `ls *`
do
cksum $file >> old_file (this is for the old files)
done 
for file in `ls *`
do 
cksum $file >> ftp_file (this is for the ftped files)
done
diff old_file ftp_file

wherever they dont match ... that one is failing. (of if they are not matching it just meas the file has been changed......
This User Gave Thanks to dazdseg For This Post:
# 5  
Old 09-01-2010
Hi methyl,

In response to your queries..
local is unix and remote is windows system and i dont have admin access to the servers.

i will be connecting from unix to windows and will do get for the files and all are binary files.(The files are bigger, takes around 45 t0 60 mins to get).

please suggest.

and also, cksum command is not working in the remote windows system.

Last edited by tarakant; 09-01-2010 at 07:41 AM..
# 6  
Old 09-01-2010
set ssh on the remote windows server, and enable the sshd services.

For example, i install CYGWIN to enable the sshd services on windows server.

then set keyless ssh, and scp these files to Unix env.
# 7  
Old 09-01-2010
Just mentioning unix and Windows without flavour or versions is just a bit too vague.

Raw "ftp" is not really suitable for this job but can work well on a good clean private LAN. Beware that many implementations will not work with files larger than 2 Gb or files with fancy filenames.
There is usually much better software than "ftp" but we must assume that you have explored all options.
Even compressing the file prior to transmission gives you a basic verification when uncompressing the file.

Verifying the transfer is more a matter of system design and one would expect a data file to contain checkable data integrity controls.

Always check "ftp" return codes in your script and have provision for timeouts and hangs. It is advisible to transfer a file under a temporary name and only rename the file if the transfer is successful.

I have had variable results when comparing Windows file sizes and unix file sizes for Binary files (it's to do with block padding). The file sizes for Text files will always disagree but the difference can be calculated.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Validate compressed files

Hi All, I have zip file that needs to be validated and checked for 5 times with sleep of 60 seconds. Some thing like below #!/bin/bash counter=1 while do curl -i -k -X GET `strings tmp.txt |grep Location| cut -f2 -d" "` -H "Authorization: Token $TOKEN" -o $zip_file ## this is... (6 Replies)
Discussion started by: Master_Mind
6 Replies

2. Shell Programming and Scripting

awk to compare files and validate order of headers

The below awk verifies the count and order of each text file in the directory. The script does execute and produce output, however the order of the headers are not compared to key. The portion in bold is supposed to do that. If the order of the headers in each text file is the same as key, then... (0 Replies)
Discussion started by: cmccabe
0 Replies

3. Shell Programming and Scripting

Validate input files daily

We have a job which we need to run on daily bases, before loading data in a table we need to validate whether the input file is received or not. Daily client will place the files in a particular path.Below files which I need to process for 04/01/2013(Load date).... (2 Replies)
Discussion started by: katakamvivek
2 Replies

4. Shell Programming and Scripting

Validate input files and update

We have a job which we need to run on daily bases, before loading data in a table we need to validate whether the input file is received or not.Inputfile formatsrc_sps_d_Call_Center_Reporting_yyyymmdd_01.dat SPS-Service nameYYYY-yearMM-MonthDD-dayLike above we will get n number of files for... (1 Reply)
Discussion started by: katakamvivek
1 Replies

5. Shell Programming and Scripting

[Validate] Script to remove files olderthan X days

Hi, I have written a small shell script to remove the files olderthan X days (say 30). But I not sure how it acts on the filesystem, as I am using rm. if then echo "##############################################" echo "Invalid no .of arguments\n" echo "Usage:\n$0 PATH... (6 Replies)
Discussion started by: karumudi7
6 Replies

6. Shell Programming and Scripting

Validate before moving files.

File Format dataline1,12,,,,; dataline2,24,,,; dataline3,12,,,,; dataline4,24,,,; COUNT=0004; Hi, I have source(/home/rgupta) and destination folders /home/rgupta/arch and /home/rgupta/err. Source folders has many files. I want to move the files from source to destination with some... (1 Reply)
Discussion started by: ravigupta2u
1 Replies

7. Shell Programming and Scripting

Help needed to validate and merge Two files

Hi.. I am new to unix !!! Please help me in combining two files by validating the first column. The files are tab limited !!! I have a file which has the following contents... DEPT1_BABYCARE 20 abce DEPT2_PHARMACY 50 cdgde DEPT3_WOMENS ... (7 Replies)
Discussion started by: ashokrajaa
7 Replies

8. Filesystems, Disks and Memory

Not able to FTP the files to a FTP server

Hi , We are facing a weird problem in our project. we need to send some xml & audio files to a remote FTP server from a Linux box, we are doing this in Perl script using Net::FTP->. Issue here is.. when FTPed the files using Perl scripts, only empty files ( 0 byte) are getting created on the... (2 Replies)
Discussion started by: kishorepotta
2 Replies

9. Shell Programming and Scripting

sftp validate files before removal

I have a shell, which locks calls an expect script that connects to an sftp and downloads all files. I need it to then check each file in my directory and compare the sizes. If the file name and file size match, then delete the file on the server. Is this possible? :confused: (2 Replies)
Discussion started by: tariqjamal
2 Replies

10. Shell Programming and Scripting

sftp shell that will validate files

Hi. I've written a shell that will sftp about 5,000 to 10,000 files a day (they are small 200 byte files). Where I'm stuck at is that I need to not only copy those files from the ftp server, I also need to delete the files on the ftp server after they have been ftped BUT before they've been... (0 Replies)
Discussion started by: biglarrrr
0 Replies
Login or Register to Ask a Question