Hotfolder with inotify-tools, loop FOR not working


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Hotfolder with inotify-tools, loop FOR not working
# 8  
Old 04-06-2018
Quote:
Originally Posted by RudiC
Where is the directory listing as requested? Did you copy all the files in one go, or one by one? By the way, why don't you mv all .xlsx files in one go?
I do not understand what you mean by directory list.
I copy all the file in one go
the command mv all .xlsx files in one go or not,
move files at once or not? I need it to be one by one.
# 9  
Old 04-06-2018
Quote:
Originally Posted by apmcd47
It is because of the way you were using inotifywait.
Code:
inotifywait -e create /some/dir

will wait for the create event in /some/dir, report that event and exit.
. . .
Yes, but shouldn't the for fullname in /HOTFOLDER/*.xlsx then run across ALL files present? Or do you think it's a race / timing problem? Why, then, would the 4th (i.e. the last) file be treated only?


Quote:
Originally Posted by pasaico
I do not understand what you mean by directory list.
ls /HOTFOLDER/

Quote:
. . . I need it to be one by one.
Why?
# 10  
Old 04-06-2018
Quote:
Originally Posted by apmcd47
It is because of the way you were using inotifywait.
Code:
inotifywait -e create /some/dir

will wait for the create event in /some/dir, report that event and exit.

You copy 10 files into that directory and only the first is processed because the script is finished (or at least the wildcard *.xlsx is expanded) before the second file is added.

Code:
inotifywait -m -e create /some/dir | while read dir event file

will run the loop for every file added to the directory /some/dir and will process the files one-at-a-time.

Andrew
Andrew,
I had already tried with the -m option but it does not work, not even a file is moved

Code:
while inotifywait -m -e create /HOTFOLDER/ ; do

---------- Post updated at 02:47 PM ---------- Previous update was at 02:33 PM ----------

Quote:
Originally Posted by RudiC

ls /HOTFOLDER/

Why?
Code:
ls /HOTFOLDER/

HOTFOLDER is empty folder


Because this is the part of code that does not work ... there's more but it's not relevant to the problem
# 11  
Old 04-06-2018
If it is an empty folder, why should ANY file be copied from there?
# 12  
Old 04-06-2018
Quote:
Originally Posted by RudiC
If it is an empty folder, why should ANY file be copied from there?
HOTFOLDER is a shared directory with samba, empty.

The files are copied into HOTFOLDER which is then processed using a script that automatically sends emails with attachments to the .xlsx file.

The problem is that the LOOP FOR stops responding when inotifywatch recognizes the event of a new file inside HOTFOLDER
# 13  
Old 04-06-2018
Quote:
Originally Posted by RudiC
Yes, but shouldn't the for fullname in /HOTFOLDER/*.xlsx then run across ALL files present? Or do you think it's a race / timing problem? Why, then, would the 4th (i.e. the last) file be treated only?
Somehow by the way pasaico is copying the files the fourth file is the first to be created. As soon as it is created inotifywait exits and there is a full ten seconds before it is run again. I imagine the wildcard expansion is faster than the copy which is why only one file is seen. When I tested the code 1.xlsx was copied.

Original script (modified for testing by me):
Code:
#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER

    while inotifywait -e create ${HOTFOLDER} ; do

    for fullname in ${HOTFOLDER}/*.xlsx; do
 
    if ! [ -d "$fullname" ]; then
            send=${HOTFOLDER}/$(date +'SEND-%F')
            sleep 10s
                    mkdir -p $send
                    mv -- "$fullname" $send
    fi

    done
    done

If we put that sleep statement between the while and for statements you give the copy time to catch up. Try it.

This is the solution I proposed (now properly coded):
Code:
#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER2

inotifywait -q -m -e create ${HOTFOLDER} | while read folder event file
do
   if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]]
   then
      fullpath="$folder/$file"
      send=${HOTFOLDER}/$(date +'SEND-%F')
      mkdir -p "${send}" && mv "${fullpath}" "${send}"
   fi
done

Now, every time a file is copied or otherwise created the while loop is run. I am checking for a CREATE rather than a CREATE,ISDIR and the file has the correct extension.

Andrew
This User Gave Thanks to apmcd47 For This Post:
# 14  
Old 04-06-2018
Quote:
Originally Posted by apmcd47
Somehow by the way pasaico is copying the files the fourth file is the first to be created. As soon as it is created inotifywait exits and there is a full ten seconds before it is run again. I imagine the wildcard expansion is faster than the copy which is why only one file is seen. When I tested the code 1.xlsx was copied.

Original script (modified for testing by me):
Code:
#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER

    while inotifywait -e create ${HOTFOLDER} ; do

    for fullname in ${HOTFOLDER}/*.xlsx; do
 
    if ! [ -d "$fullname" ]; then
            send=${HOTFOLDER}/$(date +'SEND-%F')
            sleep 10s
                    mkdir -p $send
                    mv -- "$fullname" $send
    fi

    done
    done

If we put that sleep statement between the while and for statements you give the copy time to catch up. Try it.

This is the solution I proposed (now properly coded):
Code:
#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER2

inotifywait -q -m -e create ${HOTFOLDER} | while read folder event file
do
   if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]]
   then
      fullpath="$folder/$file"
      send=${HOTFOLDER}/$(date +'SEND-%F')
      mkdir -p "${send}" && mv "${fullpath}" "${send}"
   fi
done

Now, every time a file is copied or otherwise created the while loop is run. I am checking for a CREATE rather than a CREATE,ISDIR and the file has the correct extension.

Andrew
Maybe you're right, but the strange thing is that this script worked for months.

Now anyway
I also modified your script by inserting sleep 10s before MKDIR and it works ok

Code:
#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER2

inotifywait -q -m -e create ${HOTFOLDER} | while read folder event file
do
   if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]]
   then
      fullpath="$folder/$file"
      send=${HOTFOLDER}/$(date +'SEND-%F')
      sleep 10s
      mkdir -p "${send}" && mv "${fullpath}" "${send}"
   fi
done

thanks for the suggestion and the help
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Tip: inotify cron

Dear members, moderators and others. While working on <insert project>, a need has surfaced to watch a directory, and when a file comes, to do appropriate action. So, i started writing some shell code, aware of linux inotify-tools package with inotifywait. Also, i'm seeing a lot of similar... (1 Reply)
Discussion started by: Peasant
1 Replies

2. Red Hat

Issues installing inotify-tools on RedHat Linux

I wish to install inotify-tools-3.20.1-2.4.x86_64.rpm on Linux and fire inotifywait command. Initially i was looking for "inotify-tools-3.xx.tar.gz" as instructed here http://jensd.be/248/linux/use-inotify-tools-on-centos-7-or-rhel-7-to-watch-files-and-directories-for-events I have... (9 Replies)
Discussion started by: mohtashims
9 Replies

3. Shell Programming and Scripting

For-loop not working

Hello all, I would like to unzip some files with a for-loop. Is there anyone who could tell me how I should do this - in a correct way? for file in $(ls); do echo gzip -d < $file | tar xf -; done The problem is the pipe - I believe. But how could I do it? I need it for the command... (4 Replies)
Discussion started by: API
4 Replies

4. UNIX for Advanced & Expert Users

Dynamically add paths to inotify

I have initiated an inotify process with --fromfile option and the file contain the paths to be monitored. /usr/local/maldetect/inotify/inotifywait -d -r -o /usr/local/maldetect/inotify/inotify_log --fromfile /usr/local/maldetect/sess/inotify.paths.28364 --exclude (^/var/tmp/mysql.sock)... (2 Replies)
Discussion started by: anil510
2 Replies

5. Shell Programming and Scripting

while loop not working

hi everyone i am facing a very strange problem . please help me why my while loop is not working. below is my code #!/bin/ksh file=RCMC_ABC_999_080924_210813.DAR echo ${file} value=001 count=10 echo "abc" echo ${count} value=$((${value} + 1 )) echo ${value} while ] do echo... (3 Replies)
Discussion started by: aishsimplesweet
3 Replies

6. SCO

inotify() in SCO UNIX

Hi I'm going to write a program in C language to watch activity on file system in SCO UNIX openserver 5.0.6. Apparently Linux provide an API called "inotify" , I'm wondering what's the SCO UNIX package for this ? (1 Reply)
Discussion started by: javad1_maroofi
1 Replies

7. Shell Programming and Scripting

While loop not working

here we go.. While loop is not working file.. It also invokes one more shell scripts for which parameters need to passed on. while read line do #### #### We want to have a logfile for each load #### PLog="${LogDir}/${Script}.log"; #### ... (5 Replies)
Discussion started by: premkumardr
5 Replies

8. Shell Programming and Scripting

For loop not working...! :(

Could some one help me on this... For loop is working...! for x in $i do for y in $j do z=`echo $y | awk '{print $2}'` if then FS=`/usr/bin/echo $y` echo $FS >>$Basic_location/out.csv fi done CPRT="Cpoyright @ BTOIDCIM" done (3 Replies)
Discussion started by: bullz26
3 Replies

9. Shell Programming and Scripting

if loop is not working

Hello i am trying to remove a line using an input file , but this depends upon user interaction here is the sample #!/bin/sh echo "Please enter whether you want to remove Profile" read value1 if ;then sed /movie/d temp.txt> temp3.txt else echo "Script didnot remove profile" fi ... (3 Replies)
Discussion started by: ranga27
3 Replies
Login or Register to Ask a Question