UNIX for Advanced & Expert Users

Hi everyone,

Just interested to know everyones opinions on the fastest unix compression utility with okejish compression (doesnt have to have awsome compression). I know of gzip bzip2 (sucks lol) and a couple of others but what is a great one for compression large amounts of data that wont eat to hard into the cpu and is fast.

All opinions are welcome.

Thanks.

Check out Compression Tools Compared

There are plenty names in the graphs listed to test. If it's still too slow for you, think about getting faster CPUs ^^

Thank you , Also is there a tool similar to gzip, but when you compress it, it doesnt waste the extra storage space. Because you know how when you gzip a file it keeps the original file and starts creating the gzip one and then deletes the original file when its done. Is there something that doesn't waste this kinda space?

In my eyes, this "waste" of storage is just for your safety. What would you do, if for some reason like a full filesystem, the compression would fail and you got a half eaten source file and a half baked compressed file. At least while the compression is running, you have to have that waste in my eyes.

May I ask for what you need such a fast compression tool? Maybe there is some other way you can handle it if you tell people in here, what you want to achieve or what kind of mechanism is around it.

its actually not waste of space.
Did you mean something like running the compression on the fly - using primary memory and just have what is needed at the end ?

That's kind of not being safe, where you might lose the source.

The problem with gzip is that it is still strictly single-threaded, so it won't scale on a multiprocessor system. A solution to this could be to write a script which "tar"s the source into several distinct tar-files (there should be some mechanism to balance the size of the different files) and gzip these archives simultaneously.

Regarding the "waste of space": it is possible to feed gzip directly via a pipeline:

Code:

tar -cf - <source> | gzip -9 > <destination>

should work.

I hope this helps.

bakunin

The original poster wanted to minimize CPU impact, so maybe it's not a problem that gzip would only use 1 CPU. I've found parallel bzip quite handy when I've got a big job to do and don't care about tying up >1 CPU. Its speed scales nearly linearly with the number of CPUs, and you can limit it to something less than all, if you wish.

Cheers,
Eric