Red Hat

In a manner of speaking. The longer the file, the longer the transfer takes, the longer it takes for anything processing the file to properly handle it, the more likely any sort of problem will cause the connection to break before transfer completes.

What's your network topology like between the two computers?

How long does the full-sized job take? I've seen large copies fail because the connection gets timed out after 2 say hours because an intervening firewall has blocked UDP "keep alive".

To copy that 15GB file takes about 10 hrs.

I would think if that file is 15GB that you are going to need a lot longer sleep time before the rm -f is ran.

I take it there are two processes here -- a local one which copies the files over, and another on the server, which runs unseen, and extracts the .gz files into logs.

You shouldn't be mixing and matching like that when they're both working in exactly the same folder. Either do it from one side, or the other side, not both. There will inevitably be mistakes when one or the other starts at exactly the wrong time.

If it had to be split, what I would do is this:

As an aside: Why is this running as root? All it's doing is operating on files, there's no reason to use root for that. That's dangerous.

Yes. I figured I am missing 1.5GB of data, and it would take about an hour to copy. So I put sleep 80min. I'll see result on Monday