Shell Programming and Scripting

Greetings all. I have a repository server which receives, without exhaggeration, several million files a week. The majority of these files are in .csv format, which means they're highly compressable. They are spread throughout numerous directories where there are configured monitoring utilities which parse them as they arrive. They parsers will only parse files as old as 7 days, so at day 8 I would like the files to be tarred/gzipped.

I would like to do this, and am trying to figure out the best approach. I could use find, which would go through and recursively tar/gzip everything it finds. However this doesn't move the files to a new location. Granted, I could use a second find command and move everything to a mirror of the repository directory structure.

I was also considering just using tar, creating an archive using the directory structure of the source data, setting it to remove the files, but I can't find a way in tar to remove only files older than N days old.

I was hoping I could get some suggestions on the best approach to take.

To illustrate:
Right now I have these directories:
/splunk/MSP/
/splunk/CEMP/
/splunk/CEMP/PBTS/
/splunk/CEMP/BTS/
/splunk/CEMP/cedar/
/splunk/CEMP/sbc/

and in each one there are several thousand files. Imagine there are files in each dir called a and b, where a is 8 days old, b is 1 day old.

In a perfect world I would have the script generate a single tar/gz file in a specified directory, which had the complete directory structure contained inside with only file a from each dir. After running the script, inside each directory only file b should remain in the actual repo.

Your thoughts and suggestions are appreciated.

Hi,

Simple approach would be to use find like;

Code:

find /splunk/MSP -type f -mtime +8| tar cf /afiles.tar -

Then run;

Code:

find /splunk/MSP -type f -mtime +8 -exec rm {} \;

Or you could run find with xargs options/

Or to zip

Code:

find /splunk/MSP -type f -mtime +8 | tar cf - | gzip - > /afiles.tar.gz

Regards

Dave

Moderator's Comments:

Please use next time code tags for your code and data

you can actualy zip whole dir recursed and zip will update archive with only those 8 days old .cvs files and delete them after that
so you will get 1 updating archive with all those files

If the data is important, that is a terrible approach. There's a race: a file on the cusp of -mtime +8 may be excluded from the tar list yet included in the rm list. The result would be the silent deletion of a file that was never archived.

Regards,
Alister

With millions of files the find command will take some time to complete, so invoking it 2 times sounds like a bad idea. I'd redirect the output of the find command to a file and use that to identify which files need to be processed.
This way the problem that alister addresses is solved too.