Shell Programming and Scripting

Hi ,

I have a number of files having an average line count of 10,000 lines. I have to read each file line by line and output the parsed lines into one single output file. The combined size of all the input files will be around 25 MB.
I'm using a re-directed while-read loop

Code:

while read LINE
do
.
.
. 
done < $FILENAME

But all the lines are not parsed into the output file. Some of the lines in big files containing more than 10,000 lines are being ommitted and it goes on to read next file. All the lines in files below 3000 lines are read correctly and not ommitted.

I haven't got any clue as to why this is happenening. Is it a memory issue or code issue ?
Can anyone give me an insight as to why this is happening and how can it be rectified.

This is just an example? Is your parsing really just adding a line count? How are you iterating over the files?

As is there, this is much better

Code:

awk '{ print NR, $0; }' file1 file2 ...

This is just an example. Just want to mention the skeletel of the loop. Have huge number of line parsing conditions inside the loop .

And as for awk , the body of the loop contains many back-slashes . So while converting from this loop, it gives many errors.
So I thought i would attack one at a time. First I want to rectify, the omitting of lines from reading problem.
Is it caused by the while-read loop ?

There is nothing wrong with the skeleton of the loop.

Everything else, the stuff you didn't post? I have no idea.

Post a sample of the input you have and the output you want and we'll show you how to do it.

How do you notice these lines missing? In a text editor? One thing that comes to mind is printing it to a terminal if they have \r line endings it'll seem to overwrite. Also in bash I'd use read -r
Are you using any continue or break, maybe your logic is wrong. It would certainly be easier if you posted real code.

It really depends on your input files and what you do inside the loop. The only thing that will not always work correctly with this loop skeleton is that there should be double quotes around "$FILENAME" .

Providing that these are correctly-formatted unix text files the only major stopper might be if individual records are too long or if you are using arrays (whether it be Shell or awk).