Red Hat

I repeat: This is a bad scheme to use here. Either have one end control both operations so that they never run simultaneously, or never put partial files where they might get mangled -- only put in complete ones.

I "inherited" the crontab from an admin who is no longer here. Since I am not a sysadmin, I try to avoid making too many changes on the system. As far as I know, to run scp from crontab I (as a user) have to be setup on the other machine. I could probably run as oracle or some user. I will test it.

Fair enough, but you may have inherited something rather junky. If you have to put in sleeps "in case it's not done yet", something's very wrong with it.

Sorry, I'm not sure I understand what you mean by
"I take it there are two processes here -- a local one which copies the files over, and another on the server, which runs unseen, and extracts the .gz files into logs"

My crontab runs:
20 3 * * 1,2,3,4,5,6 /data05/oradata/dpdump/cpFromDB.sh > /data05/oradata/dpdump/cpFromDB.log 2>&1

And here is my script:

cpFromDB.sh

scp -C root@xxx.xx.xxx.3:/data05/oradata/dpdump/*.gz /data05/oradata/dpdump/.
ssh -l root xxx.xx.xxx.3 rm -f /data05/oradata/dpdump/*.gz /data05/oradata/dpdump/db*.log

So, scp is the one that "a local one which copies the files over", but where is the other one that "on the server, which runs unseen, and extracts the .gz files into logs"? Is it ssh line?

No problem with running such a copy as root. We need the root ulimit for this size of file transfer.

Hmm. 10 hours to copy 15 Gb suggests something like a 4 Mbits/sec leased line rather than say a Gigabit local LAN connection. Maintaining a 100% reliable WAN connection across a leased line for 10 hours is a dream. The rsync command across a slow link is a lot better at dealing with network glitches than scp .

Have you considered a local (relatively quick) backup followed by a slow secondary backup of that backup to the remote site utilising rsync?

I was confused for a while, pardon me. I got the source and destination mixed up.

Guys, thank you for your help. Unfortunately, using "sleep " didn't help. I still get corrupted, incomplete file.

I ran ulimit -a. Looks like I do not have any size limitation.


ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 71680
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 71680
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


I'm going to read up on rsync now.