Visit The New, Modern Unix Linux Community


General compression question


 
Thread Tools Search this Thread
Operating Systems Linux General compression question
# 1  
General compression question

I am looking for an alternate solution other than gzip or bzip2 to compress files that are 3 to 4 GB each and will be hundreds per day. Aside from increasing storage anybody found a good tool?
# 2  
You have to be a bit more specific on what you need, and in what way gzip/bzip2 don't match your expectations.
# 3  
compression of large (3-4 GB each) data sets
speed = able to handle 300-400 files at once
system resources can not cripple the machine.

bzip2 works but we need to look at other possibilities
# 4  
Data compression requires a lot of CPU power. To compress several hundred files simultaneously you're simply going to need a lot of horsepower. Something like a 16 core server with 64 GB of memory is probably the minimum configuration for this task. bzip2 is best compression program I know. There are a few alternatives but they are not as good. But there no magic software that can compress hundreds of 4 GB files on a workstation in a twinkling of an eye.
# 5  
The vczip component of the Vcodex software from AT&T Labs Research might be worth looking into.

vcodex data transformation package


Depending on what your starting data looks like the algorithms in Vcodex might generate better compression than what you've used so far.
# 6  
7zip looks like it might suit your needs.

http://www.7-zip.org/download.html

7zip can use multiple cores very well, as well as using a maximum number of cores, so that will help to not bring your system to a crawl. However, this might cause a strain on your disk system.

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #537
Difficulty: Medium
An enumerated data type allows a user to define a list of keywords associated with integers.
True or False?

10 More Discussions You Might Find Interesting

1. Programming

General question about learning Python

I am planning on taking a class in Python. My choices are 2.5 or 3.0. Which version should I choose? I am getting the impression they are two separate paths. thanks. (5 Replies)
Discussion started by: djehresmann
5 Replies

2. UNIX for Dummies Questions & Answers

General question about folder level permissions

How is the level of access on a particular folder determined? I have heard (its just hearsay so am not particularly sure of it) that the access a particular user/group has to a low level directory is also affected by the level of access granted to the user/group on its parent directories. e.g. ... (1 Reply)
Discussion started by: jawsnnn
1 Replies

3. Solaris

General question about configuring DHCP on Solaris

Dears, my question is that possible to configure any server to work in solaris zone i mean any server cos i read that DHCP can't be work on solaris zone is that real or not Best Regards Alexander Corvinus (4 Replies)
Discussion started by: corvinusbsd
4 Replies

4. UNIX for Dummies Questions & Answers

General Programming Question

Experience level : New to programming in Linux. Forgive my noobiness in this context with regards to programming, language or grammar. Some Background info : I have seen a lot of programs which I use (Modo and Maya especially) which can use a command line to run the different parts of the... (2 Replies)
Discussion started by: snd321
2 Replies

5. UNIX for Dummies Questions & Answers

Executable directory general question

This may be a dumb question (but this is the UNIX for Dummies Q&A forum :) But I'm wondering, what is the purpose of a directory being given execute permissions? I can't execute a directory, only files, right? If I make a directory executable, the files inside still won't execute unless I give... (1 Reply)
Discussion started by: FredSmith
1 Replies

6. Shell Programming and Scripting

general question on executing file in drwx--x--x directory

suppose: nevermind, i figured it out, haha... (2 Replies)
Discussion started by: cul8erppl
2 Replies

7. Shell Programming and Scripting

general question?

Perl, Python, and PHP are these languages easy to use? Are they command line or are they part of a GUI? (2 Replies)
Discussion started by: wmosley2
2 Replies

8. UNIX for Dummies Questions & Answers

General Question

Hi, I've been racking my brains trying to remember, but, whats the command to change the default shell? I'm currently always in the Korn shell and I want to start out in the Bash shell. I'm running a variant of BSD I guess in Mac OS X 10.2.2 and Mandrake. Thanks. ccindyderek:confused: (4 Replies)
Discussion started by: ccindyderek
4 Replies

9. UNIX for Dummies Questions & Answers

General Proxy Question

This is quite a general question: I am trying to detect whether people accessing my network are using a proxy server. This is *not* to ensure that web pages are not cached! The only way that I can think of doing this is to intercept at packet level and examine the source port for... (1 Reply)
Discussion started by: sam_pointer
1 Replies

10. IP Networking

General Proxy Question

Firstly I must apologise; I have posted something similar in the "general UNIX newbies" forum; but in reterospect I think that it's more appropriate here. Anyhoo; I am trying to detect whether people accessing one of my servers are coming via a proxied connection. I must add that this is... (1 Reply)
Discussion started by: sam_pointer
1 Replies

Featured Tech Videos