Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

inotifywait to wait until completion

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting inotifywait to wait until completion
# 1  
Old 10-14-2011
inotifywait to wait until completion
Hi,

I am using inotifywait to monitor a directory where files are being transferred into.

I want inotifywait to tell me when a file has been completely transferred not just part of it.
I tried "create", "close" and "close_write" but it seems that inotifywait always gets triggered even if the file is only partially transferred.
Is there a way to get inotifywait to "wait" until the file has completely transferred?

I tried putting a "sleep" command to make sure the transfer is completed, but it doesn't help as inotifywait gets triggered twice. Once to tell me the file is partly transferred and a second message telling me it's completely transferred. I don't want my code to be executed twice.

Here is my code:
Code:
#!/bin/bash

inotifywait -qmre create --format '%w%f' /$PATH/data_raw/ | while read f; do

if [ -d $f ];
then
    mydir=`basename $f`;
    echo "$mydir is a new directory";
    #echo $f;
else
    #sleep 10;
    myfile=`basename $f`;
    myfilename=`basename $f .nc`;
    echo "$myfile is a new file!";
    mydir=`dirname $f`;
    echo "mydir = $mydir";
fi
done

exit 0

Thanks!


Moderator's Comments:
Mod Comment Video tutorial on how to use code tags in The UNIX and Linux Forums.
Last edited by jejeking; 10-14-2011 at 12:48 PM.. Reason: Please use code tags, thank you
# 2  
Old 10-14-2011
Here is a little simulation of what you are trying to do :

Make a file and simulate a process on it

Code:
$ touch toto
$ tail -f toto &
[1] 9854
&

Now launch an "almost" endless loop that check every 10 sec if there is a process on it :
Code:
while :
> do
> trap break INT
> if ( fuser -fu toto >/dev/null 2>&1 )
> then echo "still running"
> else echo "ended"
> break
> fi
> sleep 10
> done
still running
still running
still running
...

... so far nothing happen : the process is still on the file, the loop see it and continue its check every 10sec check.

In another tty, i do a
Code:
$ kill 9854

on the tail -f process (pid=9854) see in you first window :

Code:
...
still running
[1]+  Complété              tail -f toto
ended

When the loop detects that no more pid is on the file, it echo "ended" and breaks out the loop

---------- Post updated at 08:28 PM ---------- Previous update was at 08:22 PM ----------

Note that you can also

Code:
cp -pr remoteserv:/remotePATH/remote_file /localPATH/local_file &
wait $!
echo "copy ended"

Of course those code are poor, i don't checksum nor check size nor check return code, but this was just a simple example for training purpose
# 3  
Old 10-14-2011
Not sure I follow your reasoning...
If I understand correctly, I should check on the process associated with the transfer of the file instead of using inotifywait?
I am not an expert in this by far so I have no idea how to go about doing this. How do I know what PID is associated with each file transfer?
Isn't inotifywait a much simpler way to do this?
I thought I was just missing a minor detail in my statement.

I think I'd rather keep inotifywait instead of trying a complete different solution. I am not against it...I am just not sure how to go about doing what you're talking about.
Thanks though!
# 4  
Old 10-14-2011
Quote:
Originally Posted by jejeking
Hi,

I am using inotifywait to monitor a directory where files are being transferred into.

I want inotifywait to tell me when a file has been completely transferred not just part of it.
I tried "create", "close" and "close_write" but it seems that inotifywait always gets triggered even if the file is only partially transferred.
Is there a way to get inotifywait to "wait" until the file has completely transferred?
I doubt inotify's lying to you... It could be opening it once it's done again for a variety of reasons, like setting the timestamp. That'll cause a CLOSE_WRITE event.

Only the client, and maybe the daemon, truly know when a file's been transferred. Even when you see it close after being written to, that could just be a broken connection.

The solution, as usual, is for the client to inform you when the file is complete. Often this is done by uploading to one folder, then moving it into a 'complete' folder. Any files you see appear inside 'complete' are finished and whole.
# 5  
Old 10-14-2011
Ok!

After thinking more about it, I decided to just add an if loop that check if the filename contains "part", if it does contain "part" it exits, if it does not the script continues and works on the completed file.

Thanks all for your suggestions and ideas.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Inotifywait restart script prevents reboot when started

Hi, maybe someone could help me optimizing this little script. It works so far, but when running, reboot does not work. If kill inotifywait reboot from shell works. I think some optimization is required here. This script starts at the end of the boot process, from an external device and... (3 Replies)
Discussion started by: lowmaster
3 Replies

2. Shell Programming and Scripting

Bash-Completion, an example

Hello I've followed several bash-completion threads, usualy none were answered, because of obvious reasons. (of which i'm just aware since a day or a few) Non-the-less, because i was writing scripts, i never considered it to be possible to add this functionality. Also, because i though that... (0 Replies)
Discussion started by: sea
0 Replies

3. Shell Programming and Scripting

Help with my BASH script with WINE and inotifywait

I'm trying to improve a script that I have running on a Ubuntu desktop that basically runs a powerpoint presentaions with announcements to TV's in diffrent parts of our building. My current script uses WINE to execute powerpoint viewer. The scripts that I currently use relies on inotifywait's... (1 Reply)
Discussion started by: binary-ninja
1 Replies

4. Shell Programming and Scripting

Inotifywait tool in no there in Redhat 5.6

Hi, I want to monitor any new file to be created in some of the directory using inotifywait tool but this is not available in Redhat 5.6. Could you please let me know how to achieve (monitoring of file) without using inotifywait too? Because i dnt want to install this tool due to some reason.... (1 Reply)
Discussion started by: Pawan Kumar
1 Replies

5. Shell Programming and Scripting

Script using inotifywait

Is there a way to get my script to only trigger when a .ppt is created? #!/bin/bash while inotifywait -e create /ticker/powerpointshare; do sleep 30; sudo chmod -R 777 /ticker/powerpointshare/*.*; sleep 15; sudo reboot; done I'm not sure if this is even possible... (0 Replies)
Discussion started by: binary-ninja
0 Replies

6. Shell Programming and Scripting

calling a shell script in background and wait using "wait" in while loop

Hi, I am facing a strange issue, when i call a script from my while loop in background it doesnt go in background, despite the wait i put below the whil loop it goes forward even before the process put in background is completed. cat abc.txt | while read -u4 line do #if line contains #... (2 Replies)
Discussion started by: mihirvora16
2 Replies

7. Shell Programming and Scripting

bash completion

hello, I have been trying for a couple days to figure this out to no avail. I am converting some csh code to bash. I have converted everything except the completion code. #bashrc (I set this alias in my bashrc) alias test='source ${PATH}/test.sh' #${PATH}/test.sh (returns some aliases and... (0 Replies)
Discussion started by: platypuus
0 Replies

8. Shell Programming and Scripting

wait command - cat it wait for not-chile process?

Did not use 'wait' yet. How I understand by now the wait works only for child processes, started background. Is there any other way to watch completion of any, not related process (at least, a process, owned by the same user?) I need to start a background process, witch will be waiting... (2 Replies)
Discussion started by: alex_5161
2 Replies

9. Shell Programming and Scripting

Need to execute 2 scripts, wait, execute 2 more wait, till end of file

:cool: I need to execute a shell script to do the following: cat a file run two back ground processes using the first two values from the file wait till those background processes finish run two more background processes using the next two values from the file wait till those background... (1 Reply)
Discussion started by: halo98
1 Replies

10. Shell Programming and Scripting

file name completion

I'ld like to enable file name completion in ksh88 on AIX 5.2. My terminal is set to xterm. I've set the shell editor to emacs and ampped the arrow keys. Is there a way to map the tab key to a command like ESC= or any other file name completion command in emacvs that I may be not aware of since I'm... (1 Reply)
Discussion started by: rein
1 Replies
Login or Register to Ask a Question