Purging 2000+ directories efficiently


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Purging 2000+ directories efficiently
# 1  
Old 06-20-2013
Purging 2000+ directories efficiently

Hi

I have a requirement wherein i need to purge some directories.

I have more than 2000 directories where i need to keep data for 10 days and delete the rest. What i am looking for is an efficient way to achieve this.

There are four mount points from where i need to delete the files.

Below is an example of the structure of the mount points.

Code:
/XXXX/XXX/XXXX/XXXX/XXXX/<AAAAA>/<aaaa>/duplicate
                          exception
                          archive

/XXXX/XXX/XXXX/XXXX/XXXX/custom/xxx/<aaa>/archive

/XXXX/XXX/XXXX/XXXX/XXXX/<format>/archive

/XXXX/XXX/XXXX/XXXX/

I am aware of the conditions that should be in place for the deletion to take place but since the number of directories is too much it would be helpful if anyone can suggest an efficient way of doing this.

All values within <> eg. <AAAAA> will be fetched using sql scripts.

The shell i am using is ksh.

Last edited by jim mcnamara; 06-20-2013 at 09:28 AM..
# 2  
Old 06-20-2013
Whats your current plan in mind for this?
# 3  
Old 06-20-2013
How you get the mountpoints should be external to the cleanup command. Get sql to write the four full mountpoints to a file, called for this example
Code:
/tmp/foo.txt

or write a separate script to create the file. If you need help with that we a LOT more information.

Code:
LOGFILE=/path/to/logs/cleanup.log_$(date +%Y_%m_%d)
while read mpoint
do
    find $mpoint -type f -mtime +10  -exec rm {} \; 2&>1  >> $LOGFILE &
done  < /tmp/foo.txt
wait

This creates four parallel child processes (jobs), each cleaning up one mountpoint. It then waits until all four jobs finish.
# 4  
Old 06-20-2013
Getting the mountpoints wont be a problem and can be hardcoded to a certain extent eg. In the path
/XXXX/XXX/XXXX/XXXX/XXXX/<AAAAA>/<aaaa>/duplicate ... /XXXX/XXX/XXXX/XXXX/XXXX/ part is generic and can be hardcoded.
Using the part /<AAAAA>/, /<aaaa>/ can be fetched using a sql query so we can get and traverse into /XXXX/XXX/XXXX/XXXX/XXXX/<AAAAA>/<aaaa> path . Now inside this there
are the following dirs where purging needs to be done.

/XXXX/XXX/XXXX/XXXX/XXXX/<AAAAA>/<aaaa>/duplicate
exception archive
There are 1000+ dirs of this kind (/XXXX/XXX/XXXX/XXXX/XXXX/<AAAAA>) with the same structure as above.

What i am looking for is a way to create parallel processes which can purge these dirs parallely.

A POINT TO REMEMBER IS THAT THE NUMBER OF DIRS IS NOT CONSTANT AND IS ALWAYS INCREASING so if there are 1200 (/XXXX/XXX/XXXX/XXXX/XXXX/<AAAAA>) kind of dirs

then we should not have 3 processes cleaning up 400 dirs each.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Purging logic

Hi guys i am having following files in a directory a_07:15:13.txt a_07:16:13.txt a_07:17:13.txt a_07:18:13.txt a_07:19:13.txt a_date file will be created on a day to day basics so that it may pileup on a long go.So i need to create a logic so that the files will be gunzipped to a backup... (1 Reply)
Discussion started by: mohanalakshmi
1 Replies

2. Shell Programming and Scripting

Disc space issues and purging of files

Hi All, I am looking forward to create a unix shell script to purge the files. The requirement is: 1) Do df -k and check the current space occupied for the /a1 folder. 2) If the space consumed is greater than 90 %, delete all the DEF* files from a subfolder /a1/archive. Example: df... (4 Replies)
Discussion started by: shilpa_acc
4 Replies

3. Shell Programming and Scripting

Fine Tune - Huge files/directory - Purging

Hi Expert's, I need your assitance in tunning one script. I have a mount point where almost 4848008 files and 864739 directories are present. The script search for specific pattern files and specfic period then delete them to free up space. The script is designed to run daily and its taking around... (19 Replies)
Discussion started by: senthil.ak
19 Replies

4. Shell Programming and Scripting

purging of Files

Hello All, I want to delete the files based on the days. like, Files available under directory /abc want to delete if they are older than 15 days. Files available under directory /pqr want to delete if they are 7 days old and some files under directory /xyz should get deleted if they are... (5 Replies)
Discussion started by: ssachins
5 Replies

5. Shell Programming and Scripting

Shell script for purging the 3 days old files

Hi all, I try to write shell script to the below requirement. I have Hard coded the oratab location and take the list of databases from oratab and find out archive log locations for each database, and list more than 3 days old files for each location and purge those. ... (2 Replies)
Discussion started by: mak_boop
2 Replies

6. Shell Programming and Scripting

Error While Purging Files

find /filearchive/ -type f -mtime +7 -exec rm weblogs*.log {} \; This worked only if this comand is executed int he unix comand prompt, but when i put this in the shell script it is not recognizing the file.It says weblogs: No such file or directory Am i doing anything wrong here ? (4 Replies)
Discussion started by: svishh123
4 Replies

7. Shell Programming and Scripting

Purging a Set of Files

Hi Frineds, I want to delete a set of files which are older than 7 days from teh current date.I am totally enw to shell scripting, can anyone help me with a sample code to list out the files which are older and then remove them from the directory. Please help THanks Viswa (5 Replies)
Discussion started by: svishh123
5 Replies

8. Shell Programming and Scripting

Using xapply efficiently?

Hi all, Were currently using xapply to run multiple ssh instances that then calls a script that returns the PID of a webserver process. Currently we have like 30 xapply statements in a script call checkit which checks various webserver processes on various unix/linux boxes. My question... (0 Replies)
Discussion started by: bdsffl
0 Replies

9. Shell Programming and Scripting

Purging script

Hi, Need to change the following command to also purge the child directories after /data/tmp within one command (recursively check for X number of days old files and purge accordingly)? e.g. /data/tmp/a, /data/tmp/a/b, /data/tmp/log. find /data/tmp -type f -mtime +7 -exec rm -f {} \; ... (2 Replies)
Discussion started by: egls
2 Replies
Login or Register to Ask a Question

Featured Tech Videos