12-08-2010
How to zip/tar millions of files?
Hi guys,
I have an issue processing a large amount of files. I have around 5 million files (some of them are actually directories) in a server.
I am unable to find out the exact number of files since it's taking "forever" to finish (See
this thread for more on the issue).
Anyway, now I want to move these ~5 million files to a different location so my first thought was to tar/gzip the files and SCP them somewhere else as a single file however the tar process is also taking a loooong time to finish (in fact it never finished and I cancelled the job after 10 hours).
Basically I just want to build a single package containing the ~5 million files (zip, tar, cpio, raw data, whatever) so that I can easily move and transfer the files to a different location.
Any ideas?
Thank you.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hello,
I am trying to return the name of the resulting file from a .zip archive file using unix unzip command.
unzip c07212007.cef7081.zip
Archive: c07212007.cef7081.zip
SecureZIP for z/OS by PKWARE
inflating: CEP/CEM7080/PPVBILL/PASS/G0063V00
I used the following command to unzip in... (5 Replies)
Discussion started by: oracledev
5 Replies
2. UNIX Desktop Questions & Answers
Hi,
I want to backup my SQL database using tar zip but I'm paranoid that I will archive it. What I mean is I want the files to stay where they are but make a zipped copy of the files as well, I don't want to delete the originals.
Is the command?
tar -cvzf databasename.tar.gz... (1 Reply)
Discussion started by: chickenhouse
1 Replies
3. Shell Programming and Scripting
Hi Guru's,
I have to write a shell script which groups file names based upon the certain matching string pattern, then creates the Tar file for that particular group of files and then zips the Tar file created for the respective group of files.
For example, In the given directory these files... (3 Replies)
Discussion started by: rahu_sg
3 Replies
4. Shell Programming and Scripting
Hello,
I just saw that on my vps (centOS) my oscommerce with a seo script
has created millions of tmp files inside the /html/cache/ directory.
I would need to remove all those files (millions), I tried via shell but the vps
loads goes to very high and it hangs, is there some way to do a... (7 Replies)
Discussion started by: andymc1
7 Replies
5. UNIX for Dummies Questions & Answers
Hi,
I need a help with zip and tar. I have no done any scripts before with zip command.
What I need to achieve is list files in a directory with a specific name (ID_DATE format- given examples) and then zip (or gunzip which I need to use, I am not sure) with timestamp on the file name and then... (15 Replies)
Discussion started by: Vijay81
15 Replies
6. Shell Programming and Scripting
Need to
1. archive all the files in a directory from the previous month into a tar/gz file, ignoring all already archived 'tar.gz' files
2. Check created .tar.gz file isnt corrupted and has all the required files in it. and then remove the original files.
I am using a function to get the... (1 Reply)
Discussion started by: Prev
1 Replies
7. Shell Programming and Scripting
Hi
I have a requirement in unix shell where I need to zip multiple files on server to one single .zip file. I dont see zip command in AIX and gzip command not doing completely what I want.
One I do .zip file, I should be able to unzip in my local Computer.
Here is example what I want... (9 Replies)
Discussion started by: RAMA PULI
9 Replies
8. Shell Programming and Scripting
I need a script file for backup (zip or tar or gz) of old log files in our unix server (causing the space problem). Could you please help me to create the zip or gz files for each log files in current directory and sub-directories also?
I found one command which is to create gz file for the... (4 Replies)
Discussion started by: Mallikgm
4 Replies
9. UNIX for Beginners Questions & Answers
Hi all i am very new to shell scripting and need some help from you to learn
1)i have some log files that gets generated on daily basis example: i have abc_2017_01_30_1.log ,2017_01_30_2.log like wise so i want to zip this 4 logs which are created on same date into one zip folder.
2)Post zipping... (2 Replies)
Discussion started by: b.saipriyanka
2 Replies
10. Shell Programming and Scripting
Hi all i am very new to shell scripting and need some help from you to learn
1)i have some log files that gets generated on daily basis example: i have abc_2017_01_30_1.log ,2017_01_30_2.log like wise so i want to zip this 4 logs which are created on same date into one zip folder.
2)Post zipping... (1 Reply)
Discussion started by: b.saipriyanka
1 Replies
LEARN ABOUT CENTOS
zipmerge
ZIPMERGE(1) General Commands Manual ZIPMERGE(1)
NAME
zipmerge - merge zip archives
SYNOPSIS
zipmerge [-DhIiSsV] target-zip source-zip Op source-zip ...
DESCRIPTION
zipmerge merges the source zip archives source-zip into the target zip archive target-zip. By default, files in the source zip archives
overwrite existing files of the same name in the target zip archive.
Supported options:
-D Ignore directory components in file name comparisons.
-h Display a short help message and exit.
-I Ignore case in file name comparisons
-i Ask before overwriting files. See also -s.
-S Do not overwrite files that have the same size and CRC32 in both the source and target archives.
-s When -i is given, do not before overwriting files that have the same size and CRC32.
-V Display version information and exit.
EXIT STATUS
zipmerge exits 0 on success and 1 if an error occurred.
SEE ALSO
zipcmp(1), ziptorrent(1), libzip(3)
AUTHORS
Dieter Baron <dillo@giga.or.at> and Thomas Klausner <tk@giga.or.at>
NiH June 4, 2008 ZIPMERGE(1)