Visit Our UNIX and Linux User Community


Fine Tune - Huge files/directory - Purging


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Fine Tune - Huge files/directory - Purging
# 8  
Old 06-09-2011
yes , just try all nine in a single row and let us know the o/p.
Quote:
Originally Posted by senthil.ak
@Mirni,
By the statement from mann2719 xargs will delay the process, so shall I use the -exec flag?
@mann2719,
For Mtime +14 I'm having 9 seraching pattern, Shall i combine them in one like
Code:
-name '*(MON|TUE|WED|THU|FRI|SAT|SUN|WEEK|WEEKLY)*'

..? The file will have name like
Code:
2011_(MON)(RERUN).CSV

Also if the find has more than one search pattern is it will loop for more than once or loop for once and search for all pattern.
# 9  
Old 06-09-2011
mann1279,

The o/p is not coming like what expected, its taking lot of CPU usage also pattern has the () .
Is it *(MON)* , *(TUE)* equal to the *(MON|TUE)*
# 10  
Old 06-10-2011
Quote:
I think , by having xargs in command will add burden on tunning, since first it will add files in buffer then it will remove where as in direct command , it will keep removing once it finds the file/dir.
xargs will not burden anything. The difference between -exec and |xargs can be significant if there are a lot of arguments, with xargs being the winner. I already said this in the previous post, but let me re-iterate in more detail:
Code:
 find . -exec rm {} \;

will fork() a process for each file. If find returns a million files, you will end up with a million rm commands. This is much more expensive than doing
Code:
 find . | xargs rm

because this construct will run rm only once for many files; for how many depends on your system -- it's defined in limits.h.
Try it for yourself if you don't believe me:
Code:
$ ls | wc
  37883   37883  367719
$ time find . -maxdepth 1 -type f -exec cp {} dump \;

real    1m16.008s
user    2m0.508s
sys    0m37.818s

$ time find . -maxdepth 1 -type f | xargs cp -t dump

real    0m1.197s
user    0m0.268s
sys    0m0.712s

Newer shells also provide a '+' version of -exec, which basically does the same as xargs -- feeds the command as many arguments as it can:
Code:
 $ time find . -maxdepth 1 -type f -exec cp -t dump {} +

real    0m1.050s
user    0m0.256s
sys    0m0.660s

This User Gave Thanks to mirni For This Post:
# 11  
Old 06-10-2011
Quote:
Originally Posted by mann2719
I think , by having xargs in command will add burden on tunning, since first it will add files in buffer then it will remove where as in direct command , it will keep removing once it finds the file/dir.
No it won't, because they can run simultaneously -- where find by itself cannot.
# 12  
Old 06-11-2011
For find performance with -exec you can also use
Code:
-exec rm -f {} \+

instead of
Code:
-exec rm -f {} \;

With + you will achieve similar performance as with xargs, with minimal code change.
# 13  
Old 06-13-2011
Thanks to all, I'm finalized go with xargs and testing old/New modified script.

Old script took - 3 days 9 hours 4 minutes 0 seconds

New script i just now triggered and i will post the finding s sooner.

mann1279,

The o/p is not coming like what expected, its taking lot of CPU usage also pattern has the () .
Is it *(MON)* , *(TUE)* equal to the *(MON|TUE)*
# 14  
Old 06-15-2011
New script takes only 10 hours and 35 min 23 seconds. But it takes always 15 -21 % of CPU. Earlier its too less than 10 , so any thought on this.?but still testing on with exec + will post the final result to all.

Last edited by senthil.ak; 06-15-2011 at 07:37 AM.. Reason: missed to add.

Previous Thread | Next Thread
Test Your Knowledge in Computers #883
Difficulty: Easy
A computer security vulnerability is a weakness in design, implementation, operation or internal control.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Need help with listing file name and modified date on a huge directory

hi, We have a huge directory that ha 5.1 Million files in it. We are trying to get the file name and modified timestamp of the most recent 3 years from this huge directory for a migration project. However, the ls command (background process) to list the file names and timestamp is running for... (2 Replies)
Discussion started by: subbu
2 Replies

2. Shell Programming and Scripting

Disc space issues and purging of files

Hi All, I am looking forward to create a unix shell script to purge the files. The requirement is: 1) Do df -k and check the current space occupied for the /a1 folder. 2) If the space consumed is greater than 90 %, delete all the DEF* files from a subfolder /a1/archive. Example: df... (4 Replies)
Discussion started by: shilpa_acc
4 Replies

3. Shell Programming and Scripting

Fine tune this perl script to add router

Hi, I have this routine that reads a microsoft dhcp.netsh dump. Where it finds optionvalue 3 STRING "0.0.0.0" Replace it with the router IP based on the network !/usr/bin/perl while ( <> ) { if ( /\# NET / ) { $net = $'; $net =~ s///g; } else { s/set optionvalue 3... (1 Reply)
Discussion started by: richsark
1 Replies

4. Programming

SQL : Fine tune Insert by query

i would like to know how can i fine tune the following query since the cost of the query is too high .. insert into temp temp_1 select a,b,c,d from xxxx .. database used is IDS.. (1 Reply)
Discussion started by: expert
1 Replies

5. Shell Programming and Scripting

purging of Files

Hello All, I want to delete the files based on the days. like, Files available under directory /abc want to delete if they are older than 15 days. Files available under directory /pqr want to delete if they are 7 days old and some files under directory /xyz should get deleted if they are... (5 Replies)
Discussion started by: ssachins
5 Replies

6. Shell Programming and Scripting

Shell script for purging the 3 days old files

Hi all, I try to write shell script to the below requirement. I have Hard coded the oratab location and take the list of databases from oratab and find out archive log locations for each database, and list more than 3 days old files for each location and purge those. ... (2 Replies)
Discussion started by: mak_boop
2 Replies

7. Cybersecurity

How to fine Tune and Harden the Linux kernel

Hi, As a a security audit, how can I proceed further with Fine tuning and Hardening the linux kernel... I am not sure with the steps how to proceed further... If i do some thing wrong, then its comes with the Kernel panic error. So, I am afraid, how to do the tuning with the kernel.. (1 Reply)
Discussion started by: gsiva
1 Replies

8. Shell Programming and Scripting

Best way to diff two huge directory trees

Hi I have a job that will be running nightly incremental backsup of a large directory tree. I did the initial backup, now I want to write a script to verify that all the files were transferred correctly. I did something like this which works in principle on small trees: diff -r -q... (6 Replies)
Discussion started by: same1290
6 Replies

9. Shell Programming and Scripting

Error While Purging Files

find /filearchive/ -type f -mtime +7 -exec rm weblogs*.log {} \; This worked only if this comand is executed int he unix comand prompt, but when i put this in the shell script it is not recognizing the file.It says weblogs: No such file or directory Am i doing anything wrong here ? (4 Replies)
Discussion started by: svishh123
4 Replies

10. Shell Programming and Scripting

Purging a Set of Files

Hi Frineds, I want to delete a set of files which are older than 7 days from teh current date.I am totally enw to shell scripting, can anyone help me with a sample code to list out the files which are older and then remove them from the directory. Please help THanks Viswa (5 Replies)
Discussion started by: svishh123
5 Replies

Featured Tech Videos