looping through files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting looping through files
# 1  
Old 09-08-2010
looping through files

I am writing a ksh which has to load 7 files(.dat files) from input directory into oracle tables using sql loader. The process has to take each file at a time and once if it is loaded succesfully using sql loader into oracle tables then the process has to pick next file and load it into oracle tables. Here we do not know when we will have the files in the input directory. So my script has to check for a file, if exists then load it if it does not exists then has to wait for a period of 15 mins and then start looking for a file and so on. Below is the code which i have written. Any help is greatly appreciated.
Code:
 
while [ 1 ]
do

## check for .dat files in inputdir
FILE_EXISTS="0"
FILE_EXISTS=`ls ${INPUT_DIR}/*.txt | wc -l`
echo "value in File_exist is $FILE_EXISTS"
if [ $FILE_EXISTS -eq 0 ]; then
   continue;
fi

##  if files exists then start processing them
for file in `ls ${INPUT_DIR}/*.txt`
do
   execute_sql_loader $file
   RC=?
   echo "sqlldr generated output $RC"
      if [ $RC -ne 0 ]; then
         continue;
      fi
done

sleep 10;
done


Last edited by vpv0002; 09-08-2010 at 10:48 PM..
# 2  
Old 09-08-2010
please indent your code so its easier to read.
# 3  
Old 09-09-2010
This is the way I'd approach it:

Code:
#/usr/bin/env ksh

if ! cd $INPUT_DIR
then
        echo "unable to switch to $INPUT_DIR"
        exit 1
fi

need=7                 # number of files needed before we are finished
while (( $need > 0 ))
do
        for file in *.dat          # for all .dat files currently in the directory 
        do
                echo "processing: $file"
                execute_sql_loader $file
                rc=$?
                if (( $rc > 0 ))
                then
                        echo "error loading file: $file rc=$rc"
                else
                        echo "file loaded successfully: $file"
                fi


                mv $file $file.processed        # prevent finding it again; maybe delete it instead?
                need=$(( $need - 1 ))         # one less file needed
        done

        if (( need > 0 ))
        then
                echo "$(date) waiting 15m before making next pass; need $need files"
                sleep $(( 15 * 60 ))                    # wait 15 minutes before trying again
        fi
done

This code is completely untested so there might be a typo or something I missed. You might also want to check to ensure that you don't process the same file twice, assuming that you won't get duplicate file names.

Your code said .txt, but your description said .dat -- I assumed the later.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Looping through files in pairs

Hi all, Please guide. It has to do with parsing the input file names. I have a fairly large number of files, I want to do some operations on them in a pairwise fashion (every file has a pair). The names are in the following pattern, with the pairs of files named with _1 and _2 , the... (4 Replies)
Discussion started by: newbie83
4 Replies

2. Shell Programming and Scripting

Looping through Files

Hi all , I am new on this forum . I have to face a particoular implementation issue and I need some help . Requirement : I need to read a particoular file (an xml file) and after reading it I need to call an Oracle Stored Procedure passing the content of the file as paramenter , in order... (3 Replies)
Discussion started by: Kolas79
3 Replies

3. Shell Programming and Scripting

Not looping or creating files

So my script is supposed to repeat for every server in my file, but as of now it is getting stuck on my awk commands # Read file cred.txt (with one IP per line), connect to servers (one at a time), and download directory listing i=1 param=$(sed -n "{$1}p" $parm_dir/cdm_param.txt) #Get the last... (6 Replies)
Discussion started by: MJCreations
6 Replies

4. Shell Programming and Scripting

Conditional Looping In Files

I have a req. where i need to read data from multiple files and take counts of row which satisfy the condition. e.g.: FILE1: Col1 Col2 Col3 12 ab cd 15 de fg 25 gh tm FILE2: Col1 Col2 Col3 21 ab1 cd1 13 de1 fg1 25 gh1 tm1 --- --- FILE-N... i need to find the count of rows... (6 Replies)
Discussion started by: kunal007
6 Replies

5. Shell Programming and Scripting

looping through files with different extensions

Hi all, I am trying to make a for loop invoking files with different extensions (*.ugrd and *.vgrd) and I cant just make it work. Cant figure out how to load the files so as to use them in subsequent commands like the ones in this pseudo code. the files are arranged such that in one date for... (8 Replies)
Discussion started by: ida1215
8 Replies

6. Shell Programming and Scripting

Looping through 2 files simultaneously

Hi all, I'm having a problem with a script which should ultimately provide a filename by reading a value from file1 and file2 then join together. I'm planning to use a loop/ loops to get the values out of both files and create a single string unfortunately the code currently treats the second... (7 Replies)
Discussion started by: chris01010
7 Replies

7. Shell Programming and Scripting

Looping through files...

I posted this in the Solaris forum, but I don't think it's platform specific, so I'm posting it here. Here is the situation. We are a company that has been using a professional publishing system, the software is called "ProType". It runs on Solaris 2.4, however it is no longer supported and we... (6 Replies)
Discussion started by: Fred Goldman
6 Replies

8. Shell Programming and Scripting

Looping on a list of files...

This isn't working for multiple files. It works for one file though. exists1=$(ls | grep gspp*) for FILES in $exists1 do echo "Loading $exists1" ... (23 Replies)
Discussion started by: lazerfoursix
23 Replies

9. Shell Programming and Scripting

Help looping through files, please...

Okay... I've solved one problem. Here's the next. I'm writing a script file that needs to go through a directory and list all files in that directory. I'm using TCL/TK. I figured out how to go through the directory and how to loop through it, but I ran into a little problem. ... (2 Replies)
Discussion started by: kapolani
2 Replies

10. Shell Programming and Scripting

looping files

Hi, I have a file a.lst which lists all files. as a.dat b.dat c.dat I want to process these files mentioned in the list file in a loop. Say I want to display only the first line of all the files a.dat , b.dat, c.dat. How can I go about it? Please help. (5 Replies)
Discussion started by: dharmesht
5 Replies
Login or Register to Ask a Question