UNIX for Advanced & Expert Users

I have a directory that need to backup, its size is about 150GB consist of multiple files and directories. I try to compress it become a single archive file using these commands:

Code:

tar cjf this_archive.tar.bz2 this_archive/

or

Code:

tar cf - this_archive/ | 7z a -si -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on this_archive.tar.7z

both commands always take a long time to process compressing the directory, even if it can finish the process, the output file archive always corrupt/error.

I want to know if there's any efficient compressing method/technique for large size directory that capable to do archiving directory like in my case?

What Operating System and version to you have and what Shell do you use?

Are any of the individual files larger that 2 Gb ?
What is the size of the largest file?

What backup software and backup medium do you use for your normal day-to-day backups?

- I'm using RedHat Enterprise Linux 5.5, Kernel 2.6.18-194.el5 (default), with 16Gb RAM.
- No
- Largest file is ~100Kb, and average size is ~50Kb
- I create relatively simple shell script with those tar/7z commands, scheduled run by crontab, if it can successfully create the archive file then it'll will push to raid5 storage on my NAS server.

You should update the archive with only the files that have been changed between runs. Appends new files. Try not to remake the entire archive every time. Read the man page for tar, consider multiple volumes.