UNIX for Dummies Questions & Answers

Ummm.... depends on what you mean by that. It would be more accurate to say that it's deleting the original copies. So the end result is like a move operation. This is what the OP requested. I hope he uses another techique to back up the os.

Guys, many thanks for your help here, it really is appreciated. I will have a go with your suggested work around.
And no we don't edit the script each day I was just filling a few parameters in to make it a bit more readable.
Thanks again I will post back the results to let you know how it goes.

Sorry can you just explain the line -

(cd $Archive ; find . -type f )| xargs rm

I can see that you have copied the files into $Archive, along with its sub-directories. But then this line seems to be deleting everything in $Archive, ie what you have just created? Shouldn't it be looking in /big/file/collection and doing mtime +7 again??? Hopefully I have misunderstood. Also if you did that then potentially you could miss a file that was created between the copy and the delete?
Thanks.

Whoa, I've never seen someone argue both for and against a change to one of my scripts before. No, it should not be looking in /big/file/collection. And, yes, if it did that, it would be broken, but not precisely for the reason you gave.

You want to get the list of names to delete from the Archive directory. You do not want to go back and get a second list of names from the main directory. If the files was copied to the Archive directory, it is clearly safe to delete the original. I hope that you are not creating files with older timestamps. But you sense a race condition and you're right. A file that was something like 6.999 days old during the first find might be 7.001 days old during the second find. Getting the list of files to delete from the Archive directory is the only secure thing to do. And it will be faster anyway...fewer files to look at; and no need to check timestamps.

When you put a command in parentheses, it is run in a subshell. That subshell will change the cwd to Archive and execute the find. Meanwhile, the parent shell continues to have it's own (unchanged) cwd. The xargs command inherits this cwd from the parent shell and passes the cwd on its offspring, which are the rm processes. So the rm processes will be affected by the original "cd /big/file/collection" while find command is affected by the "cd $Archive".

Maybe this is what was bothering dangral too. Using parentheses to spawn a subshell is a standard technique that goes back even to the old bourne shell in the 70's.

Yes, I mistakenly did not take the subshell into account.

Ah, I see. That makes sense. Fantastic it could just do the trick. I will give it a whirl and report back.
Many thanks once again.

Excellent, it has certainly improved things. I ran a test with a subset of data, the old script processed it in 210 minutes. The new script with the above code in it ran in 70 minutes - fantastic.
Many many thanks for your help.