Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

Zip million files taking 12 hours or more

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Zip million files taking 12 hours or more
# 1  
Old 03-11-2016
Wrench Zip million files taking 12 hours or more
Hi

I have task to zip files based on modified time but they are in millions and it is taking lot of time more than 12 hours and also eating up high cpu
is there any other / better way to handle it quickly with less cpu consumption
Code:
find .   ! -name \"*.gz\" -mtime +7  -type f | grep -v '/.*/' | xargs gzip

Thanks
RL
Last edited by rbatte1; 03-11-2016 at 06:02 AM.. Reason: Converted ICODE tags to CODE tags and corrected spelling
# 2  
Old 03-11-2016
This should speed up the process :
Code:
cd <your dir>
find . ! -name \"*.gz\" -mtime +7 -type f -exec gzip {} \+
cd -
# Or 
find /absolute/path  ! -name \"*.gz\" -mtime +7 -type f -exec gzip {} \+

As for CPU usage, what does high cpu usage means ? All the cores are utilized fully or ?

Hope that helps
Best regards
Peasant.
# 3  
Old 03-11-2016
What is the -name \"*.gz\" supposed to do?
What is the | grep -v '/.*/' supposed to do?
Please give example input files and state the ones to be discarded.
Moreover, -name "*.gz" looks safe and is more common, but is different from -name \"*.gz\".
Last edited by MadeInGermany; 03-11-2016 at 08:38 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help 'speeding' up this 'parsing' script - taking 24+ hours to run

Hi, I've written a ksh script that read a file and parse/filter/format each line. The script runs as expected but it runs for 24+ hours for a file that has 2million lines. And sometimes, the input file has 10million lines which means it can be running for more than 2 days and still not finish.... (9 Replies)
Discussion started by: newbie_01
9 Replies

2. Shell Programming and Scripting

How can we Zip multiple files created on the same date into one single zip file.?

Hi all i am very new to shell scripting and need some help from you to learn 1)i have some log files that gets generated on daily basis example: i have abc_2017_01_30_1.log ,2017_01_30_2.log like wise so i want to zip this 4 logs which are created on same date into one zip folder. 2)Post zipping... (1 Reply)
Discussion started by: b.saipriyanka
1 Replies

3. UNIX for Beginners Questions & Answers

How can we Zip multiple files created on the same date into one single zip file.?

Hi all i am very new to shell scripting and need some help from you to learn 1)i have some log files that gets generated on daily basis example: i have abc_2017_01_30_1.log ,2017_01_30_2.log like wise so i want to zip this 4 logs which are created on same date into one zip folder. 2)Post zipping... (2 Replies)
Discussion started by: b.saipriyanka
2 Replies

4. UNIX for Dummies Questions & Answers

Deleting a million of files ..

Hi, Which way is faster rm -rf /path/ or find / -name -exec rm {} \; and why? (7 Replies)
Discussion started by: cain82
7 Replies

5. UNIX for Dummies Questions & Answers

Pls. help with script to remove million files

Hi, one of the server, log directory was never cleaned up. We have so many files. I want to remove all the files that starts with dfr* but I get error message when I use the *. rm qfr* bash: /usr/bin/rm: Arg list too long I am trying to write this script but not working. ... (4 Replies)
Discussion started by: samnyc
4 Replies

6. Shell Programming and Scripting

Zip Multiple files to One .zip file in AIX system

Hi I have a requirement in unix shell where I need to zip multiple files on server to one single .zip file. I dont see zip command in AIX and gzip command not doing completely what I want. One I do .zip file, I should be able to unzip in my local Computer. Here is example what I want... (9 Replies)
Discussion started by: RAMA PULI
9 Replies

7. AIX

ZIP multiple files and also specify size of zip file

I have to zip many pdf files and the size of zip file must not exceed 200 MB. When size is more than 200 MB then multiple zip files needs to be created. How we can achieve this in UNIX? I have tried ZIP utility but it takes a lot of time when we add individual pdfs by looping through a... (1 Reply)
Discussion started by: tom007
1 Replies

8. Shell Programming and Scripting

how to list files between last 6 hours to 3 hours

Hi Frens, I want to list some files from a directory, which contains "DONE" in their name, i am receiving files every minute. In this i want to list all the files which are newer than 6 hours but older than 3 hours, of current time i dont want my list to contain the latest files which are ... (4 Replies)
Discussion started by: Prat007
4 Replies

9. UNIX for Dummies Questions & Answers

unzip .zip file and list the files included in the .zip archive

Hello, I am trying to return the name of the resulting file from a .zip archive file using unix unzip command. unzip c07212007.cef7081.zip Archive: c07212007.cef7081.zip SecureZIP for z/OS by PKWARE inflating: CEP/CEM7080/PPVBILL/PASS/G0063V00 I used the following command to unzip in... (5 Replies)
Discussion started by: oracledev
5 Replies
Login or Register to Ask a Question