Shell Programming and Scripting

We have shell script (in ksh) which reads the records from a csv file line by line and does some operation. We have below command to read the file (CSV has the absolute path of files stored on a server something like SER/IMP/TEST/2010/12/123465.1).

Code:

P_FILES=`cat $P_BATCH_FILE`
for i in $P_FILES
do
   P_FILEPATH=`echo $i`
   P_FILE=`basename "$i"` #to get file name with extension
   export FILE_NAME=`echo "$P_FILE" | cut -d . -f 1`
   ---then do some operations like insert the entry into database etc
done

The CSV file has around 2000 lines of records. The issue we are seeing is - the script is not reading all the records. It reads about 1190 records and comes out of the loop. We are not sure why it is not reading all lines. We did confirm that the file is correct and there is no special characters in it.

Is there a restriction on CAT command in unix? Does it read only a limited number of lines from a file? Is there a fix for the above issue? What other approach can be used for the above requirement?

Thanks so much for any help you can provide on this.

Thanks,
Shreevatsa

You have just discovered the size-limit of shell variables on your system. As the output gets large, backticks run out of space. This is one of the many forms of useless uses of cat, too.

Better form, and suitable for files of any size, less likely to strip all spacing out of the input, and more efficient in general is thus:

Code:

while read LINE
do
        do stuff with "$LINE"
done < filename

It reads the file line by line as required instead of sticking it all in memory at once.

It also obeys IFS, so you can split apart records with zero processes created instead of two:

Code:

while IFS="." read FILE EXT
do
        echo $FILE
done < filename

Thank you so much for the reply.