I'd like to handle incoming (uploaded) files from a shell script, ideally without busy polling / waiting (e.g. running a cron task every 15'). Is there a command that would just sleep until a new entry has been created in a directory, allowing scripts such as the following:
The "watchdir" command should ideally only wake up when the new entries aren't being used (or at least written to) by anyone, so the file can safely be handled. This will have to run on HP-UX, but I am interested in linux alternatives, or even pointer to the underlying API or system call.
The way I see it, I'm afraid there is no simple workaround for your problem. There is no such function not even in the glibc I guess. To do something like that, you would have to go through every processes' file descriptors (in /proc/onepid/fd for Linux) and check if there is no reference to one file inside that $SOMEDIR of yours.
Nowadays you shouldn't have to worry about those things because programs don't just write to files directly. The way to do it, is writing a temporary file and then rename() it to the location you want. rename() is an atomic function (will never corrupt the file).
In shell scripting, you could use
every minute to see what files were changed in the last minute.
Anyway, wait for more replies, there could be something that at least would get you close to what you want.
When transferring files with whatever protocol and you keep the original date for them, I guess you won't find a difference with "find -mtime".
Maybe a bit more reliable is to make find on all files and pass them with exec or xargs to "cksum", write it to some file and then compare it every interval you'd like.
So if there is any difference, may it be a different sum or a new sum for a file that didn't exist already or a removed one, you should notice that.
Last edited by zaxxon; 08-15-2008 at 10:37 AM..
Reason: It's cksum, not chksum
When transferring files with whatever protocol and you keep the original date for them, I guess you won't find a difference with "find -mtime".
Maybe a bit more reliable is to make find on all files and pass them with exec or xargs to "chksum", write it to some file and then compare it every interval you'd like.
So if there is any difference, may it be a different sum or a new sum for a file that didn't exist already or a removed one, you should notice that.
Your solution is more reliable but I just agree because it's possible to modify modification times to fool find -mmin. But otherwise, I think uploading protocols do modify modification date when uploading files (even if overwriting). Maybe I misread what you meant.
Linux used to have something like FAM which would alert you when a file was altered. This is being replaced by other mechanisms which are slightly less platform-specific, but there is nothing which is stable and standard just yet. If yoiu have a dbus system, I think that includes this facility, but the name escapes me. There are discussions on this site about such systems so searching along the lines of fam and dbus might help you find stuff you like.
I didn't thought on modification on purpose. Just if you transfer files via scp and don't use the -p flag to preserve times and modes, you will have different times on the files. With -p you keep it like on the source.
But I have to admit that it depends solely on what baldyeti wants or if it is important at all for him.
I didn't thought on modification on purpose. Just if you transfer files via scp and don't use the -p flag to preserve times and modes, you will have different times on the files. With -p you keep it like on the source.
But I have to admit that it depends solely on what baldyeti wants or if it is important at all for him.
Good point. I was not thinking about scp uploads. Baldyeti didn't specify the protocol so I assumed ftp or web upload (p.e. php) which generally modify mtime.
HI
I need some help on this below one
Have summuary file coming on daily basis with list of file names and count ,if the summuary file not exist ,pool it for every 5 mins till it arrives .Once arrived remove first and last line from the file and check all the files in respective director all... (12 Replies)
Hello All,
I have a script that checks a log file. This will be checked periodically lets say every 5 minutes. I need to get new entries and process these new entries
What do you think is the best strategy here? all i can think of currently is to backup the old file and do diff. But future... (9 Replies)
Hi Guys,
we are planning to enable polling target on our SG in reference to this link
MC/ServiceGuard: Subnet Goes Down when one of the Cluster Node is Down - HP Customer Care (United States - English)
since our server is already in production can we apply the changes with ex. "cmapplyconf... (1 Reply)
Hi!
Im trying to make a program that polls a mysqlDB every 5 minutes.
the poll checks a scheduletable in the dB and exicutes a function if
there was a "go" in the scheduletable, and if it retrives a "no go" from the table
it should just wait for another 5 minutes before re polling the DB,
... (1 Reply)
I plan to run a script which will run in background and at predefined times send mails to user . I dont have acees to autosys/ Cron / at jobs
My loop works like this
while (true)
do
getx_time=`date +%H%M`
if ; then
script1.sh
mail_sent_flag=1
elif ; then
script2.sh... (1 Reply)
Hi,
We have a filesystem mystery on our hands. Given:
2 machines, A and Aa.
Machine Aa is the problem machine.
Machine A is running Ubuntu, kernel 2.6.22.9 #1 SMP Wed Feb 20 08:46:16 CST 2008 x86_64 GNU/Linux. Machine Aa is running RHEL5.3, kernel 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38... (2 Replies)
I have 20 or so files that are ftp'd each day to a particular directory. I want to create a script that will tell me if any of these files fails to arrive or if any additional files arrive. I'm thinking I'd have a list of file names that should arrive each day, and the script would check each days... (2 Replies)
Hey guys,
I am working on a Cshell script and I am stuck on this one part. I need to be able to copy in files to my directory but give them different names so they don't overwrite each other. For example, my folder already contains FILE.1 I want my script to name the next file copied over... (5 Replies)
Hi all,
I have a directory where some process is keeping incremental/new log files. I need to code a program which will periodically poll this directory and if it founds a new file arrived then copy that new file to some other specific directory.
We are OK with this polling approach. I just... (3 Replies)