Difference in file sizes being copied by scp. HELP!

 
Thread Tools Search this Thread
Operating Systems Linux Red Hat Difference in file sizes being copied by scp. HELP!
# 1  
Old 06-07-2012
Difference in file sizes being copied by scp. HELP!

Hello All,

I am transfering a gzipped file from LINUX to LINUX using scp -C comand.
It is a nightly job, called by crontab. After copy finishes, the file sizes are different between source and destination. Say .gz file is 14782805941 bytes on source and 13496172544 bytes on destination. When I gunzip it I get an error message "unexpected end of file'.
However, when i type EXACTLY the same command from prompt, scp runs ok and both files are the same size.
I run md5sum check against both files.
Everything is ok when I start scp manually, and I get corruption when same command is ran by crontab.

What could that be?? I really really need to fix it because it is actually the database backups, and untill I fix this problem i have to database backup!

Thanks in advance!
# 2  
Old 06-07-2012
Did you check without enabling the compression i.e without the -C switch with SCP? Can you post the crontab entry for this?
# 3  
Old 06-08-2012
Is it a root cron?
# 4  
Old 06-08-2012
To admin_xor:

Here is my entry:


Code:
20 3 * * 1,2,3,4,5,6 /data05/oradata/dpdump/cpFromDB.sh > /data05/oradata/dpdump/cpFromDB.log  2>&1


cpFromDB.sh

Code:
scp -C root@xxx.xx.xxx.3:/data05/oradata/dpdump/*.gz /data05/oradata/dpdump/.
ssh -l root xxx.xx.xxx.3 rm -f /data05/oradata/dpdump/*.gz /data05/oradata/dpdump/db*.log

And I am going to test without the compression right now.

Thanks!

---------- Post updated at 10:26 AM ---------- Previous update was at 10:08 AM ----------

Quote:
Originally Posted by methyl
Is it a root cron?

I would think so, since the job is running as root.

Last edited by Scrutinizer; 06-08-2012 at 12:25 PM.. Reason: code tags
# 5  
Old 06-08-2012
Have you considered rsync? More checks and balances than using scp.
# 6  
Old 06-08-2012
(Only asked about root in case there was a different ulimit).

Have you tried putting a say sleep 30 between the scp and the ssh ? I'm serious (the second command deletes the source file and the slave process could still be running).

I agree that the compression is pointless on a compressed file. Also that rsync is preferred if both computers support the command.
# 7  
Old 06-08-2012
I just ran a test, I used cron to SCP a file twice - with and without the -C switch. Both runs were ok, and the file sizes are equal. However, this test file was much smaller than the real one. The real one was 15GB, while the test file was only 30MB.

So, could it be that the size of a file is causing a problem?

I might be able to run the large file only over the weekend, other wise it is getting in a way of our backup process.

---------- Post updated at 11:58 AM ---------- Previous update was at 11:54 AM ----------

Quote:
Originally Posted by methyl
(Only asked about root in case there was a different ulimit).

Have you tried putting a say sleep 30 between the scp and the ssh ? I'm serious (the second command deletes the source file and the slave process could still be running).

I agree that the compression is pointless on a compressed file. Also that rsync is preferred if both computers support the command.

------------
I will put sleep in the code. It will run tonight, and result I will see Monday morning. It could be a very good catch, deletes before the previous process finishes.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Cp via NFS vs. scp yields unexpected difference

I have two Linux machines, Linux1 and Linux2. They both have two NFS mounts. We'll call them /scratch1 and /scratch2. And they both reside on the same NetApp filer. If I copy a 512Mb file from /scratch1 to /scratch2 while logged on Linux1 it takes 40s. However if I scp this file from... (1 Reply)
Discussion started by: crimso
1 Replies

2. Shell Programming and Scripting

Wait for file to get copied in server path and then proceed

Hi, I have a requirement to create below script: Script must run infinitely in background. It will check a particular type of file to be copied in specific folder of server Script must wait till any file gets fully copied ..(important) and then It will read that file Experts please... (4 Replies)
Discussion started by: Vikash163
4 Replies

3. Linux

Difference in binary file sizes after deploying package via yum/rpm

Hi Guys, I have an issue whereby I package up an rpm and when I deploy it, the binary filesizes differ from what I packaged up from the SOURCE directory. from source - -rwxrwxr-x 1 user group 685156 Sep 26 15:50 cybAgent.bin after install - -rwxr-xr-x 1 user group 683388 Oct 16 11:10... (4 Replies)
Discussion started by: gazza-o
4 Replies

4. Shell Programming and Scripting

no of files copied by scp command

hi, I have a script which copy files by using scp command. when i run that script, i can see the process meter but I want to know how many files have been copied. suppose, by using scp 10 files is being copied and I can see process meter for all 10 files. but i also want to see ... that 10... (0 Replies)
Discussion started by: anshu ranjan
0 Replies

5. Shell Programming and Scripting

Extract a number from a line in a file and sed in another copied file

Dear all, I am trying to extract a number from a line in one file (task 1), duplicate another file (task 2) and replace all instances of the strings 300, in duplicated with the extracted number (task 3). Here is what I have tried so far: for ((k=1;k<4;k++)); do temp=`sed -n "${k}p"... (2 Replies)
Discussion started by: mnaqvi
2 Replies

6. UNIX for Dummies Questions & Answers

Using sudo scp -r can't get everything copied though

I want to copy a folder and all its contents of ~700GB from a computer to another. I've tried sudo scp -r directory/ admin@host:directory but the result is that the copied folder is about 2GBs smaller than the original. I checked the shell for errors and found some "file doesn't exist" errors.... (2 Replies)
Discussion started by: MJH
2 Replies

7. Solaris

Check copied file

Hi all, If i wanted to copy file within different folders or different servers, how do i determine the copied file is absolutely correct :confused: Is it using cmp and chksum command enough? Anyway that i can make further checking? Thanks in advance for reading & anyone who reply the... (7 Replies)
Discussion started by: beginningDBA
7 Replies

8. UNIX for Dummies Questions & Answers

Can A File Be Copied to an RS232 Port on Unix?

We have a text/csv file (4mb) sitting on a Unix box. The PC belongs to a retail store and they're asking me to bring the file over to Windows so they can import the file. Can a file be copied to an rs232 port? If so how? The idea is that I'll hook up a null modem cable to the Unix com port,... (2 Replies)
Discussion started by: alivebyscience
2 Replies

9. Shell Programming and Scripting

How to check file is being copied

I wanna check whether a file is being copied ,i.e the file size is increasing. Thanks for your help (2 Replies)
Discussion started by: anhkeen
2 Replies

10. UNIX for Dummies Questions & Answers

How to find File copied completely or else ...

In Unix, I am having one file getting copied to some directory. Which command will help me ensure, that file is not completely copied to the disk? (2 Replies)
Discussion started by: videsh77
2 Replies
Login or Register to Ask a Question

Featured Tech Videos