UNIX for Advanced & Expert Users

We have seen an issue whereby every morning around the same time , we see files being deleted from /users/$userid .

We have many crons and processes running across 40+ different servers .
Possibly some rogue process is doing this .

How can one isolate the process removing stuff from the users home .

The users home is mounted from all the 40 machines : Thus which machine is running the rogue process ( and what that is ) is the challenge to figure out .

All machines are Linux boxes .

Any pointers to audit scripts/tools would be appreciated .

It is probably a process designed to clean up old files. Since it works across users, it has to have permissions. If we assume the files in question are 755 permissions, this means it has to run in a crontab owned by a user that is allowed su or sudo. Or is root to start with.

To verify this assumption, consider setting 000 permisions on some dummy files in the directory trees in question. Use the same naming conventions, ownerships, file sizes and file times as the ones that got clobbered. If the dummy file(s) are removed it has to be a priviliged user.

What this does is limit the number of crontabs to read. If the assumptions are good start grepping the crontabs for jobs that run at the time in question, limit the search to just privileged crontab files.

If all users have cron access, then you may have a script that a weak programmer wrote that everyone has shared. Good luck with that.

If you have linux consider, the inotify family. http://linux.die.net/man/7/inotify You can interactively watch for deletions in the directory trees in question.

How is the user's home mounted? Are you using an automounter?

What files disappear? Old files, new files, all files, random files? One user, some users, all users?
If you can determine the rules by which files are disappearing and the time of the process, that will go a long way towards finding the process.
When you get close, you could create files using "touch" with timestamps a few minutes apart and use this to determine what time the process runs.

Is the danger time in working hours? Is the user logged in at the time? I'd certainly look at /etc/profile and the user's own Shell profile.

One blind guess would be to look for a script running from cron at around the right time which contains a "find" statement on /users where the username parameter is not set. Similarly look through /etc/passwd for an account with home directory of /users (I've seen this one cause havoc before).

[Jim] We have all our crons run from the same userid , and its this userids home which is seeing strange removal . We also see this early in the morning around 6:00 : meaning we have a 10 minute suspect window . We are going to log 'ps -ef' in a loop for 10 minutes on a few servers where we have most of our crons to locate the process .
Thanks for the inotify tool .

[Methyl] I guess there is an automounter used to mount the users home ( we have 40+ machines ) .
Many files disappear : what hits us the ssh , and bin/prompt : We have many tools run with this userid ( and many ssh'es to our several servers which are impacted by this )
We have looked at the crons but dont see a suspect . As mentioned above , we are now going to look at raw ps output in that 10 minute window .

I was hoping that there would be some way to monitor a directory to such an extent that one could know what process ( and from which host its run ) does something to a directory ??
( We know the userid ,the directory and we know the approx timeframe - need to find the process and machine )

What does this statement mean?
What does the problem look like?

Have any files ever re-appeared?