Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to check the files existence inside a directory.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to check the files existence inside a directory.
# 8  
Old 10-30-2017
Quote:

Its not like that, I am taking everyone suggestion.
I must need to catch files, which fly in very frequent time from directory.
---------- Post updated at 12:35 PM ---------- Previous update was at 12:30 PM ----------

Code:
#!/bin/sh
DATE=`date +"%d-%m-%Y-%H:%M:%S"`
stream=S1
FLAG=FIRST
path=/bigpfstest/INVESTIG/S12
CNT=0
while [[ $CNT -le 10 ]]
do
                if [[  -f $path/*.RTM ]]
                                then
                                                for  files in `printf "%s\n" $path/*.RTM | tail -3`
                                                                do
                                                                                TT=`awk -F ',' 'NR==1{print $1}' $files`
                                                                                FILENAME=`ls  $files | cut -d '/' -f5`
                                                                                TIMESTAMP=$( date +'%H:%M:%S' -r $files)
                                                                                TRANS_TIME=$(date -d @$(printf '%.0f\n' $TT) | awk '{print $4}')
                                                                                TIME_LAG=$(date +%H:%M:%S -ud @$((`expr $(date -u -d "$TIMESTAMP" +"%s") - $(date -u -d "$TRANS_TIME" +"%s")`)))
                                                                                if [ "${FILENAME}" == "" ];
                                                                                then
                                                                                                                echo "" >/dev/null
                                                                                else
                                                                                                                echo "${DATE} ${stream} ${FLAG} ${FILENAME} ${TIMESTAMP} ${TRANS_TIME} ${TIME_LAG}" >> /bigpfstest/DPI_INVESTIG/check.csv
                                                                                fi
                                                                done
                                ((CNT=100))
                else
                                sleep 2s
                                (( CNT++ ))
                                echo "$CNT"
                fi
done


kindly help what i can change on it, to catch the file apply the for loop to do some calculation and exit.

If not found atleast it iterate for counter to keep looking for files till no. of counter reach

---------- Post updated at 01:39 PM ---------- Previous update was at 12:35 PM ----------

Requriment:

Say
Directory contains 1000 files in every 5 mins and fly quickly to other place.
I need a help to correct my script which can keep checking the directory for the files after every 10 seconds for 20 times.

If anytime it found the files inside the directory within this 20 times, execute the for logic of my script part and with if condition echo the result.

Kindly guide.

Code:
#!/bin/sh
DATE=`date +"%d-%m-%Y-%H:%M:%S"`
stream=S1
FLAG=FIRST
path=/bigpfstest/INVESTIG/S1
CNT=0
while [[ $CNT -le 10 ]]
do
                if [[ ! -f $path/*.RTM ]]
                                then
                                                for  files in `printf "%s\n" $path/*.RTM | tail -3`
                                                                do
                                                                                TT=`awk -F ',' 'NR==1{print $1}' $files`
                                                                                FILENAME=`ls  $files|cut -d '/' -f5`
                                                                                TIMESTAMP=$( date +'%H:%M:%S' -r $files)
                                                                                TRANS_TIME=$(date -d @$(printf '%.0f\n' $TT) | awk '{print $4}')
                                                                                TIME_LAG=$(date +%H:%M:%S -ud @$((`expr $(date -u -d "$TIMESTAMP" +"%s") - $(date -u -d "$TRANS_TIME" +"%s")`)))
                                                                                if [ "${FILENAME}" == "" ];
                                                                                then
                                                                                                                echo "" >/dev/null
                                                                                else
                                                                                                                echo "${DATE} ${stream} ${FLAG} ${FILENAME} ${TIMESTAMP} ${TRANS_TIME} ${TIME_LAG}" >> /bigpfstest/DPI_INVESTIG/check.csv
                                                                                fi
                                                                done
                                ((CNT=100))
                else
                                sleep 2s
                                (( CNT++ ))
                                echo "$CNT"
                fi
done

Last edited by sadique.manzar; 10-30-2017 at 03:32 PM..
# 9  
Old 10-30-2017
Code:
if [ -r job.pid ]
then
    echo $(cat job.pid) did not terminate 
    mail -s "$(cat job.pid) did not terminate" admin
    exit
fi
echo $$ >job.pid
list=$(ls *.job)
if [ $list <> "" ]
  then 
  for file in $list
  do
    fuser $file
    if [ $?  -eq  1 ]
    then
       echo $file $(date) >>job.log
       your stuff
       mv $file done
    fi
   done
fi
echo $(cat job.pid) finished at $(date) >>job.log
rm job.pid

Generally I create a requests directory to hold the input files, and and a done directory to hold the completed input files. If a file is in use, it is left for the next invocation. Create a cron job to run this as frequently as you need.
Add a line in one of your rc commands to remove any job.pid files on system startup.
You need to stop the process that removes the files from the directory and replace it with this process otherwise you will never guarantee that you have processed every file. Also with this many files being added you will need to create a dedicated incoming directory, and have a cron job delete and recreate this directory on a regular basis.
Last edited by jgt; 10-30-2017 at 05:37 PM..
This User Gave Thanks to jgt For This Post:
sadique.manzar
# 10  
Old 10-30-2017
Hey jgt,

I cannot hold the files in another directory as per my requirement.

I want last 5 files and on every file only 1st record.
I need file creation time too, so if i copy these to another location then the file creation time will change on the latest directory.

each files contains record where from 1st record of each file i am selecting
a column having time data,

If you notice on my script, echo last three attribute is file creation time, time inside record and time lag---which i am calculating from file creation time and time inside record of each files says column 1st.

kindly let me know, if more clarification required.

My for loop is must required for my requirement.
I need a script where my for loop fit.
so that if directory doesn't contain files it should not go to for loop and keep in loop till it found the files.
if it find then only it should run the for loop.
# 11  
Old 10-30-2017
Quote:
Directory contains 1000 files in every 5 mins and fly quickly to other place.
I need a help to correct my script which can keep checking the directory for the files after every 10 seconds for 20 times.

If anytime it found the files inside the directory within this 20 times, execute the for logic of my script part and with if condition echo the result.
You have to explain how these files arrive? What process assigns the file name? and what process removes the files from the directory.
How large are these files?
# 12  
Old 10-31-2017
Quote:
Originally Posted by jgt
You have to explain how these files arrive? What process assigns the file name? and what process removes the files from the directory.
How large are these files?
From destination server these files landing into my environment directory.
Through shell script assign the files to our landing directory, so every 5 mins the files keep on coming to our landing directory and from their it fly to different server within frame of time.

Once its landed ,it moved to different server.

Each file size is almost 256KB.

My requirement is i need to catch last 3 files and perform for loop of my script echo values and exit.

And if it doesn't able to find then every 10 secs it shud check for the files.
Once found perform rest of the logic and make flag change and script successful exit.

So my requirement is now,

Check the directory for file exists.
If found do for loop logic
Else
Sleep for few secs
And again check for files keep on looping

For certain no of count say 20 count each iterate after every 10secs of sleep.

If within this count not found script should exit successfully.
# 13  
Old 10-31-2017
Quote:
Once its landed ,it moved to different server.
From the time that you start your process to find the last three files and the time that you try to actually read the file how will you know that the file is still on your system?
Can you predict the file names?
My opinion is that without access to the code that either sends the files or removes the files you will not be able to accomplish your requirement with any degree of certainty.
You may be able to find a solution using a packet sniffer.
# 14  
Old 10-31-2017
Consider it stay there for 10 mins.

No of files are good in amount like 1000.

So I can catch them within that frame.

Manually i am able to do so, because i am keep looking for files in that directory.

But i dont want to try babysitting.

Just let me know how i can check for the files exist in a directory.
Let me try for 20 times to check it with sleep of 10 secs

If within that frame it catches fine else
Exit

I just need to catch once in that 20 times. If got then execute the rest of the code and exit.

Can i achieve it ?

Using flag, while, until, if any of this logic which can help me to get that.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Check if files inside a text file are found under a directory

Hi all, Please somebody help me with this: I want to check if the files listed in a text file, are found under a directory or not. For example: the file is list_of_files.txt, which contains inside this rows: # cat list_of_files logs errors paths debug # I want to check if these... (3 Replies)
Discussion started by: arrals_vl
3 Replies

2. Shell Programming and Scripting

Check the Files existence

Hi I have a requirement to check whether the files exists, then it will call other steps in shell script. I did ls *.csv|wc -l if then checking the count of the files should be more than 1 then it will call other steps. I am getting the error that too many arguements as there n... (13 Replies)
Discussion started by: cnrj
13 Replies

3. Shell Programming and Scripting

How to check whether directory has files in it or not in shell script?

hi, I am having script in which i want to check if directory has any file in it or not. If directory contains a single or more files then and only then it should proceed to further operations... P.S.: Directory might have thousand number of files. so there might be chance of getting error... (4 Replies)
Discussion started by: VSom007
4 Replies

4. Shell Programming and Scripting

Check for the existence and add them from 2 different files

Hi, I have two files file1: ALEX DANY GARY TOM MARY HARRIS file2: ALEX 3 ALEX 5 ALEX 0 ALEX 1 ALEX 0 DANY 2 (2 Replies)
Discussion started by: Diya123
2 Replies

5. Shell Programming and Scripting

check existence of files in a folder

Hi I am having a problem to verify existence of files. I need to know whether or not files in a folder that begins with a name. For example all files that start with The_File_ *. I was doing it this way, but gives me error. if text -f /work/The_File_* then ... else .. fi (5 Replies)
Discussion started by: Rodrih92
5 Replies

6. Shell Programming and Scripting

Check existence of a number of files and call other scripts

Hi, I am new to unix scripting and am jus getting to learn about it.. I need to know on how to check for the existence of a number of files in a path..i.e the files are ftp'ed from several other servers, should check if all the files have arrived, if not wait till they arrive..can i use a flag... (5 Replies)
Discussion started by: yohasini
5 Replies

7. Shell Programming and Scripting

script to check for existence of file (or else sleep for x time)

Hi Forum. I have a script that accepts 3 input parameters (source directory, list file text, sleep time) and checks for the presence of files. If not there, script goes to sleep for certain amount of time provided by 3rd input. list file text contains 1 entry but may contain more (file... (13 Replies)
Discussion started by: pchang
13 Replies

8. UNIX for Dummies Questions & Answers

Zip files inside the directory, but not the directory itself

Hi, Im facing a problem that im stucked, I have the following structure: thales@pereirtc-vbox:/home/VfARM$ ls code config doc lib manifest.bak manifest.rel manifest.v3 ns pub if i try to execute zip -q -o arm.zip VfARM/* it will create a zip file with the folder VfARM.... (2 Replies)
Discussion started by: Thales.Claro
2 Replies

9. Shell Programming and Scripting

shell :: check directory existence

Hi All, I have shell script and I need to check if some directory exist. I'm don't have the information if that directory is written in upper case or lowcase or mixed. Is there anyway to check the existence of that directory by ignoring case senestive? Thanks (3 Replies)
Discussion started by: Alalush
3 Replies

10. Shell Programming and Scripting

check for FILES existence

hi, I have a list of filenames and I want to verify if they all exist. I know "if filename" would do the trick but how do I go about a list of files? thanks (3 Replies)
Discussion started by: mpang_
3 Replies
Login or Register to Ask a Question