Visit Our UNIX and Linux User Community


<AIX>Problem in purge script, taking very very long time to complete 18.30hrs


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting <AIX>Problem in purge script, taking very very long time to complete 18.30hrs
# 1  
Old 08-16-2009
<AIX>Problem in purge script, taking very very long time to complete 18.30hrs

Hi,

I have here a script which is used to purge older files/directories based on defined purge period. The script consists of 45 find commands, where each command will need to traverse through more than a million directories. Therefore a single find command executes around 22-25 mins approximately Smilie. The entire script runs approximately 18.30hrs Smilie which is very very huge. Could someone suggest a better idea/logic to achieve this purpose and also reducing the execution time as much as possible. Thanks!

Last edited by sravicha; 08-19-2009 at 02:10 AM..
# 2  
Old 08-16-2009
Sunday morning guesswork (with by far not enough files available for serious testing, based on experience with a similar but smaller problem though):

- First, I'd separate the finding from the killing, i.e. generate a shitlist (of data to be removed) to be processed as a background task of lesser priority.

- Second, I'd rewrite thus reduce the number of searches by means of (more) regular expressions, maybe even using 'ls' and 'grep' instead of 'find' (?).
# 3  
Old 08-16-2009
Thank you for responding dr.house, and yes I did think about the option of separating the search logic and redirect them into a dump file. Later pick the entries from the dump file and proceed to delete/purge. But this one too had a negative impact on the size of dump file created(very huge), and the filesystem reached 100% before the search was completeSmilie.

Honestly speaking, I am very weak in shell scripting especially involving complex logicSmilie. It should be grateful if you could post the script with the logic you suggested. Thanks!

Last edited by sravicha; 08-16-2009 at 05:45 AM..
# 4  
Old 08-16-2009
Put together all those lines where rule is same, use find -o option to put together name rules using one find.

There is also meny different kind of finds, some days old or one year old. Think carefully how often you really need to run different kind of finds+rm. Make more than find script file, run once in month, run once in ...

Example something like this:
Code:
find . \( -name '*(MON)*'  -o -name '*(TUE)*' \) -type f

# 5  
Old 08-16-2009
Quote:
Originally Posted by kshji
Put together all those lines where rule is same, use find -o option to put together name rules using one find.

There is also meny different kind of finds, some days old or one year old. Think carefully how often you really need to run different kind of finds+rm. Make more than find script file, run once in month, run once in ...

Example something like this:
Code:
find . \( -name '*(MON)*'  -o -name '*(TUE)*' \) -type f

Thanks kshji, seems to be quite a good suggestion. If I group together all the search patterns having common purge period, I am left with only 13 find commandsSmilie. If the find command with multiple search patterns also run for approximately 22-25 mins, your logic would drastically reduce the execution time. Let me try it and post a reply.

One quick question, may I precede using "-type f" before "-name"Smilie
# 6  
Old 08-16-2009
Quote:
One quick question, may I precede using "-type f" before "-name"
The answer is yes
# 7  
Old 08-16-2009
Quote:
Originally Posted by methyl
The answer is yes
Thank youSmilie

Previous Thread | Next Thread
Test Your Knowledge in Computers #308
Difficulty: Easy
The first computer bug was created by a broken keyboard input.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script takes too long to complete

Hi, I have a lengthy script which i have trimmed down for a test case as below. more run.sh #!/bin/bash paths="allpath.txt" while IFS= read -r loc do echo "Working on $loc" startdir=$loc find "$startdir" -type f \( ! -name "*.log*" ! -name "*.class*" \) -print | while read file do... (8 Replies)
Discussion started by: mohtashims
8 Replies

2. Shell Programming and Scripting

Wget takes a long time to complete

Hi, I wish to check the return value for wget $url. However, some urls are designed to take 45 minutes or more to return. All i need to check if the URL can be reached or not using wget. How can i get wget to return the value in a few seconds ? (8 Replies)
Discussion started by: mohtashims
8 Replies

3. Shell Programming and Scripting

While loop problem taking too long

while read myhosts do while read discovered do echo "$discovered" done < $LOGFILE | grep -Pi "|" | egrep... (7 Replies)
Discussion started by: SkySmart
7 Replies

4. UNIX for Dummies Questions & Answers

ls is taking long time to list

Hi, All the data are kept on Netapp using NFS. some directories are so fast when doing ls but few of them are slow. After doing few times, it becomes fast. Then again after few minutes, it becomes slow again. Can you advise what's going on? This one directory I am very interested is giving... (3 Replies)
Discussion started by: samnyc
3 Replies

5. Solaris

How to find out bottleneck if system is taking long time in gzip

Dear All, OS = Solaris 5.10 Hardware Sun Fire T2000 with 1 Ghz quode core We have oracle application 11i with 10g database. When ever i am trying to take cold backup of database with 55GB size its taking long time to finish. As the application is down nobody is using the server at all... (8 Replies)
Discussion started by: yoojamu
8 Replies

6. UNIX for Dummies Questions & Answers

Job is taking long time

Hi , We have 20 jobs are scheduled. In that one of our job is taking long time ,it's not completing. If we are not terminating it's running infinity time actually the job completion time is 5 minutes. The job is deleting some records from the table and two insert statements and one select... (7 Replies)
Discussion started by: ajaykumarkona
7 Replies

7. UNIX for Dummies Questions & Answers

gref -f taking long time for big file

grep -f taking long time to compare for big files, any alternate for fast check I am using grep -f file1 file2 to check - to ckeck dups/common rows prsents. But my files contains file1 contains 5gb and file 2 contains 50 mb and its taking such a long time to compare the files. Do we have any... (10 Replies)
Discussion started by: gkskumar
10 Replies

8. Red Hat

login process taking a long time

I'm having a bit of a login performance issue.. wondering if anyone has any ideas where I might look. Here's the scenario... Linux Red Hat ES 4 update 5 regardless of where I login from (ssh or on the text console) after providing the password the system seems to pause for between 30... (4 Replies)
Discussion started by: retlaw
4 Replies

9. UNIX for Dummies Questions & Answers

fetchmail taking long time to fetchmail...

Hi peeps, We are having around 60 users. The time set to retrieve the mail is 300 sec. But it's taking around 1 hour to deliver mails. I am using debian sarge 3.1. any clues? And how it will affect if I decrease the time? My machine has got 1 p4 3.0 GHZ processor and 1 GB ram. The home... (2 Replies)
Discussion started by: squid04
2 Replies

10. Shell Programming and Scripting

shell script takes long time to complete

Hi all, I wrote this shell script to validate filed numbers for input file. But it take forever to complete validation on a file. The average speed is like 9mins/MB. Can anyone tell me how to improve the performance of a shell script? Thanks (12 Replies)
Discussion started by: ozzman
12 Replies

Featured Tech Videos