Shell Programming and Scripting

So, I have a filesystem with an ever growing number of files in it.

Multiple users are able to drop new files into this area and I need to know what's changed (a granularity of month by month is ok, more accurately would be better).

There are however a few issues, I can't just use the timestamps of the files as some of the timestamps are way in the past (and with some whacky users, way in the future).

I don't mind changing the timestamps and I'm also at the point where I know that all historical information is lost, I just want to know where I am going forward.

So one method I'm trying currently to implement is first of all a blanket timestamp change of everything to 11:11:11 on the 1st of the month.

This is easy enough, and will give me a baseline.

But then I need to run something (probably via cron) that once a day will go through the filesystem and find any files that don't have a timestamp of 11:11:11 on the 1st of ANY month and set them to 11:11:11 on the 1st of the CURRENT month.

Once I have a known set of timestamps for all files I can just use find to list out the changed files easily.

So the first question has to be... is there a better way to do this?

...and the second one is if so, let me know, if not... any suggestions?

Thanks!

check the find command and see if yours has the -newer option. if so, touch an empty marker file before you reset all the timestamps and then use something similar to following.

Code:

find /filesystem -xdev -newer /filesystem/marker_file

but then if then timestamps are marked prior to the last reset at first of the month, you might not get the desired results. although this is something you should be able to test now to determine.
HTH

Yeah, the problem with that is that in the future users could drop files onto the filesystem which have timestamps that bear no relation to reality (old files, new files, files with timestamps in the future or any time at all).

So searching for "newer than" or "older than" (which would be quite easy) breaks down pretty quickly.

Hence wanting to set the timestamps to something artificial (11:11:11 on 1st of a given month) to allow me to filter between what I already know and everything else.

Unless I'm missing something in your suggestion?

Thanks for the thoughts so far though!

or create a file with all known filenames as of the last scan, and next month create a new file and compare for differences. this would depend if users are using same or different filenames from month to month.

this month

Code:

find /filesystem -xdev | sort >/tmp/filesystem_month_$(date +"%m")

next month, using 04 and 05 to indicate previous and current month numbers.

Code:

find /filesystem -xdev | sort >/tmp/filesystem_month_$(date +"%m")
comm -13 /tmp/filesystem_month_04 /tmp/filesystem_month_05

although, depending on the size of the filesystem, this could be done more frequently, adjusting the date variable accordingly.

Hmm, having just run a few tests, that'll work.

I'd really been trying to do this without having to keep loads of historical text files laying about if I could (hence my focus on timestamps)...

This is a 20TB filesystem, about to scale up to 40TB in the next few months, so as I'm sure you can imagine, keeping the full text files of the listings each time could itself consume quite a bit of space. Especially if I squeeze the granularity down to daily.

But your suggestion seems to be the easiest way to get me started at least.

Thanks again... any other suggestions are of course welcome!

---------- Post updated at 03:51 PM ---------- Previous update was at 12:02 PM ----------

So, I've continued poking at this and I'm nearly at a solution which still involves the magical timestamps.

It's a bit of a hack, but it doesn't involve creating any extra files.

However I'm stuck at the final hurdle, it explodes horribly if the file names have spaces in them.... Any suggestions to get me to the final solution?

Code:

find /var/filesystem/ -xdev -type f -printf '%T+ %p \n' | grep -v 11:11:11 | xargs touch --no-create --date=11:11:11

This works fine as long as the file names don't have spaces in, but as soon as that happens, it all goes wrong

Ideas?

xargs options to help with spaces in filenames reported here.
https://www.unix.com/shell-programmin...me-spaces.html

Still no joy with those, they don't seem to make any difference. But I'll keep hacking about for other solutions.

I've modified it slightly from the last iteration:

Code:

find /var/filesystem/ -xdev -type f -printf '%T+ %p \n' | grep -v 11:11:11 | cut -c32- | xargs touch --no-create --date=11:11:11

Edging closer, will continue to examine xargs options with whitespace in them.

---------- Post updated at 04:40 PM ---------- Previous update was at 04:24 PM ----------

The problem is that I'm already using -printf to get the date format out so I can grep on it... and -print0 doesn't take arguments for formatting...