What is the best way for a script to run to monitor a directory for the presence of files and then perform a function afterwords? I was hoping to have it continually run and sleep until it detects that files are present in the directory, then break out of the loop and go on to the next step.

What it does it waits for files to be transferred to it's local directory, then when the files are present it sftp's them out to another host. I already have the sftp script working just need the proper syntax for a loop to go to sleep until it detects the files in the directory. Also the file names and conventions will constantly change so I was hoping more for a 0 byte count compared to anything else besides a 0 byte count for a condition.This is a linux box using bash as the shell.

thanks in advance guys.

Classic mailbox problem. One problem you have is: how do you know the files are really there, as opposed to partly there? Also, after you sftp them out, do you remove them?

Without addressing these problems (above), here's a simple script:

One problem you have is: how do you know the files are really there, as opposed to partly there? Also, after you sftp them out, do you remove them?

Hello, thanks for the quick response! You have good points and this has been discussed. We do remove them after transfer. another note is that although the names will change as will the byte size, they will always end in *.dat. One of other problems is as you mention what if the files are detected before they are finished transferring to the host, before they are sent back out?

thanks

What is your if statement actually doing?

thanks

what about using the watch command?

thanks

Useful for "eye-balling" the status of a process or whatever. But: No way to trigger a command (automatically) upon a condition, and Do you really want to sit there and watch for files to come in? If you do, then :

is fine.

Never. You'd have to kill the process. It's equivalent to removing the while loop and putting the script as a cron job that runs ever 5 minutes.

It's running the pipeline of ls and grep. The grep is looking for any line. If ls finds no files, it outputs no line, and the grep search fails. If the grep search fails, it exits with 1 which (in BASH logic) means false, and so the if condition fails. If the grep finds at least one line, this means there are files and so the if condition succeeds and the THEN portion is executed.

Are there other files in the directory besides these?

There are at least four solutions to this. In the worst case, you can use what is suggested by ddreggors. Here are solutions to try:

1. Move-after-write. Modify (or configure) the process that places the incoming file. When creating the file, it names the file in a distinctive way (different extension, prefixed with ., different path, etc). After closing, it moves / renames the file in a way your script expects.
2. Keep control-log file. Modify (or configure) the process that places the incoming file. After closing the file, it appends the name of the file to a control file, kept inside the mailbox folder (as "something.ctl"). Your script will rename this file, and then read it for a list of file names to sftp.
3. Use flock. This will only work IF the process, which places the incoming files, uses flock(2) on files that it creates. Use the script to use flock (shell command) to each file before it is moved.

ddreggors solution is a bit resource-intensive, and VERY linux-specific, but it will work if the uploading process does not close/reopen the file in between writes AND if it removes the file after failure (if the upload process is interrupted and the file is not completely transferred). It will fail if there are any other benign processes reading the incoming files. Here is a re-write of that solution which is more efficient (ie, doesn't use additional forks):

Nice rewrite and great points but I thought I might mention that in the quote above you make it seem like it will NOT work if "the upload process is interrupted and the file is not completely transferred". While that is not what one would want the process will still work, you will just end up with a blank or partially written file. What I mean to say is that the loop will still run and do what it is meant to do but it will preform the actions on a file that is not intact. This would happen with ANY process as far as I know. If the transfer fails then you have a bad file and no way to tell without looking inside manually (vi, nano, etc...) or diff against the original (which you might not have).

Now on the other hand, it would be a very nice feature if the upload/transfer process failed and the file was removed (as you already mentioned).

As to the Linux specific code... my bad, I am from a Linux world and should remember that nix* means *more* than Linux usually.

I also make the assumtion that this user wants a quick easy to create solution. I base this on very little knowledge granted but there seems to be an air of "quick dirty hack" (not to be taken as a bad thing, just a real world thing) written in the initial question.