Shell Programming and Scripting

Dear Gurus,
I have an application that sends me thousands of small files.
I need to write a script that sleeps for some time untile enough files have arrived then concatenate (n) files into 1 big file, then send that big file to a program to process it.

For example, if I have the following files
f1, f2, f3, f4, f5

assuming n=3
the script will do noting until f1, f2, f3 arrive, then:

It would do something like this:
cat f1 f2 f3 > bigfile and then executes another program to execute bigfile

then it would wait for the next 3 files to be available or a time period has elapsed (to cover for the case of only x < n files ever arriving) - In this example, it would wait some time after f5 arrives, then does something like this:
cat f4 f5 > bigfile and then executes another program to process bigfile

In short I need to write a script that deals with input files as a FIFO queue. I am sure some one some where must have written a similar one.

Could you point me to some source?

Thanks.

Here I give u a backbone to work on...Though I have not tried it... but should work with changes as per your need.

Code:

#!/usr/bin/ksh

#N is the number of files, you wish to wait for.
N=3

#T is time you wish to wait for in seconds.
T=10

#SPECIFIED_TIME time you assume all files have arrived.
SPECIFIED_TIME=2000

START_TIME=`date '+%Y%m%d%H%M%S'
FLAG=0

while 1
do
find $FILE_LOCATION -name $NAME* -exec ls -ltr {} \; > tmpfile
FILE_CNT='wc -l tmpfile | awk '{print $1}'`

if [ $FILE_CNT -ge $N -o $FLAG -eq 1 ]; then
  rm -f bigfile
  for i = 1 to $N
  do 
    file_nm=`head -1 tmpfile | awk '{ print $9 }'`
    sed -n '1d' tmpfile > tmpfile
    cat $file_nm >> bigfile
  done
  
  if [ $FLAG -eq 1 ]; then
    exit
  fi
  
fi

sleep $T

CUR_TIME=`date '+%Y%m%d%H%M%S'
diff=`expr $CUR_TIME - $START_TIME`
if [ $diff > $SPECIFIED_TIME ]; then
 FLAG=1
fi

done

Hi,
I need a help from any of you guys.

Actually my scenario is same but little bit different.

Iam having 3 Processes Pr1,Pr2 and Pr3 . I would like to run every process in every 15 Min and i want to load the files.
i.e
first
15 Min : Pr1 Should trigger
after next 15 min : Pr2 Should run
then next 15 min : Pr3 Should run
then next 15 min : again Pr1 should run before running it should check the status of Pr1 of the first cycle whether it is completed or not.If it is completed then it should run the Pr1 Process else it should check wheter the Pr2 and Pr3 Process are running or not. Basically the free process has to kick off.

Basically i use to get files in every 15min , so i want to process it through the above mentioned 3 Processses.

kindly help me on the same.


Script:


wait $var_band1

if [ $? -eq 0 ]

then

echo " Execute Band1 " &

var_band1=$!

echo "var_band1 : $var_band1"

sleep 3

echo "Waiting for the Next Hr Files "

else

echo "Still the Band1 Process is Running"

fi



wait $var_band2

if [ $? -eq 0 ]

then

echo " Execute Band2 " &

var_band2=$!

echo "var_band2 : $var_band2"

sleep 3

echo "Waiting for the Next Hr Files "

else

echo "Still the Band2 Process is Running"

fi



wait $var_band3

if [ $? -eq 0 ]

then

echo " Execute Band3 " &

var_band3=$!

echo "var_band3 : $var_band3"

sleep 3

echo "Waiting for the Next Hr Files "

else

echo "Still the Band3 Process is Running"

fi