UNIX for Advanced & Expert Users

looking for solution to replicate 1.5GB files to a remote location...

Currently, this process looks like the following:

move 1.5GB files into a staging area.
compress files.
rsync files to remote server.
remove compressed files.


I have performed some timings, and compress seems more efficent over gzip.

Is there an efficient way to move and compress in a single step?
wondering how to improve process listed above...
thanks

lzop might be better for compression of enormous files than gzip or compress. It's compression ratio isn't as good but it's lightning fast.

I often do this sort of thing but over a local area network with udpcast, when I replicate whole drives with it:

Code:

# Source computer
dd if=/dev/hda | lzop -c | udp-sender
# Recieving computer(s)
udp-reciever | lzop -d | dd of=/dev/hda

but udpcast uses multicasting, which won't go across the internet.

I occasionally use my own hacks, tcp_pipe or faf to handle piping through sockets across the internet but these are, well, hacks that I don't reccomend one use in a production environment. If there are any "normal" equivalents to these programs, I don't know them. Anyone?

compressing the file appears tobe more efficient then gzip, but does not compress as much:

# time compress bigFILE.arc

real 2:29.3
user 1:46.6
sys 23.2

# ll
-rw-r----- 1 root sys 549384653 Sep 20 10:10 bigFILE.Z

1566427136-549384653
1017042483 amt. compressed
1017042483/149 sec

6825788.47 per sec


gzip file:

# time gzip bigFILE

real 5:02.2
user 4:21.2
sys 22.5
# ll
total 778816
-rw-r----- 1 root sys 398751664 Sep 20 10:10 bigFILE.gz

1566427136-398751664
1167675472 amt. compressed
1167675472/302 sec

3866475.07 per sec


But when sending them across the pipe, my timings show an improvement when the files are uncompressed (using rsync):

receiving file list ...
1 file to consider
bigFILE.Z
549384653 100% 539.48kB/s 0:16:34 (xfer#1, to-check=0/1)

sent 42 bytes received 549569234 bytes 551499.52 bytes/sec
total size is 549384653 speedup is 1.00

real 16m35.83s
user 0m10.71s
sys 0m30.54s



bigFILE
1566427136 100% 2.08MB/s 0:11:58 (xfer#1, to-check=0/1)

sent 42 bytes received 397960981 bytes 552340.07 bytes/sec
total size is 1566427136 speedup is 3.94

real 12m0.30s
user 0m56.81s
sys 0m28.56s



So now, I'm leaning towards sending files uncompressed, and performing a gzip at the remote box...
thanks,
manny

gzip has a great option....you should experiment with it. you can set the compression rate with gzip with switches...

for ex.
gzip -9 file

-9 indicates the highest level of compression. I think you can choose from 1-9. It is possible to compress a gigabyte of text into like 10 megs. So maybe you can deal with the time of compression and catch up on time in terms of transit. ftp of 10 megs is waaaaay faster than a larger 1 gb file.You will have to experiment and see for yourself.

-S

You could compress to stdout and pipe it to ssh/rsh and redirect to a file at the other side.

Currently leaning towards rsync:

since rsync performs compression in-flight, it likes uncompressed files:

stats on sending compressed file:
(your mileage may vary )

551499.52 bytes/sec
total size is 549384653 speedup is 1.00

real 16m35.83s
user 0m10.71s
sys 0m30.54s


uncompressed file:

1566427136 100% 2.08MB/s 0:11:58 (xfer#1, to-check=0/1)

sent 42 bytes received 397960981 bytes 552340.07 bytes/sec
total size is 1566427136 speedup is 3.94

real 12m0.30s
user 0m56.81s
sys 0m28.56s


Maybe the best both worlds?
rsync uncompressed file, and gzip on remote box where duration and load is not an issue...

your input is appreciated