Shell Programming and Scripting

Hi,

The requirement is below:

There are 5 files in a unix directory and i need to pick up latest file from the directory that i can do like this

Code:

ls -lrt | tail -1

and this file can be processed. now the actual requirement is some times what will happen the second third...on files i.e(

Code:

ls -lrt | tail -2, -3...

) did not processed(may missed to process some times). Now what i required is the loop has to loop and check for all the 5 files and compare with the already processed files and if any file was not processed(missed file) has to pick and process next run.

Please provide some looping solution compare with already processed list.

Many thanks in advance.


Thanks,
Praveen

Code:

ls /directory > tmp.tmp   # get five files
awk ' FNR==NR {arr[$0]++; next}
        { arr[$0]++}
        END{for(i in arr) {if(arr[i]==1 {print i}  }}  ' tmp.tmp  mylist_of_files

tmp.tmp has the names of the five files, mylist_of_files has the names of all processed files.

BTW: how are your sure the files you want to process are completely written?
IF a process is writing them the file will show up in ls, it just will not necessarily have all the records it is supposed to have.

When you have file lists you also have to be sure about how they are "named" in the file. The above script
will treat /directory/file100 and file100 as completely different files.

If the file listfile contains the files you want to process (sans directory), then the post-processing could be something like:

Code:

targetdir=/.../directory
watchdir=$targetdir/.watchdir
listfile=/.../listfile

cd $targetdir
mkdir -p $watchdir

while sleep ${delay}; do
  for file in $(< ${listfile}); do
    if [[ ! -e ${watchdir}/${file} -o ${file} -nt ${watchdir}/${file} ]]; then
        touch -r ${file} ${watchdir}/${file}
        process ${file}
    fi
  done
done

But as Jim_McNamara rightly pointed out, processing a file that is being written too is the path to sadness - you have a very good chance of reading an incomplete file.

What is creating the source files?

When 'parsing' files that have chances that they are still beeing written,
i would fallback on checking their last change (using stat), and only if the change is older than <insert reasonable timeout> i'd start reading it.

The timeout could be anything from 5 secs to 5 hours.

hth

Or you can wait until the file is closed, something like:

Code:

lsof +r filename > /dev/null
process filename

Or logrotate could be used.

There are many many ways you can get this done, depending on the type of file and how it is being written.

Can additional details be provided?