Shell Programming and Scripting

Awk doesn't mysteriously stop working inside while loops, no.

I can only guess what your friend meant, but I'm assuming the lesson intended was "Don't run awk 30,000 times to process 10,000 lines". And that is certainly correct, not to mention an error I see new shell programmers making a lot. They see awk used to get a column and think "Oh, that's what awk is for", and use it everywhere. Things like ...

Code:

while read LINE
do
        FIRSTCOLUMN=`echo $LINE | awk '{print $1}'`
        SECONDCOLUMN=`echo $LINE | awk '{print $2}'`
        THIRDCOLUMN=`echo $LINE | awk '{print $2}'`
        SOMETHINGELSE=`echo $LINE | sed '...'`
done < inputfile

...are extremely silly. Whenever get asked 'why is my script slow', they're doing that.

Better way to do it would be

Code:

while read FIRSTCOLUMN SECONDCOLUMN THIRDCOLUMN REST
do
...
done < inputfile

so you run awk 0 times instead of 30,000, and sed 0 times instead of 10,000...

But I'm using awk to process entire files. Nothing silly about that.

Nice Corona688, what you said makes sense especially if you are processing large files. How about if there are many columns like says 10 columns, what is the efficient way of doing that instead of declaring 10 variables?

Supposing the columns are separated by , there's a few things you can do.

I like the set -- feature a lot, it's a simple thing which exists in any Bourne shell, capable of replacing arrays for many purposes and actually simpler to use.
This should work in any shell:

Code:

OLDIFS="$IFS"
IFS=","

while read LINE
do
        # Set the $1, $2, ... variables to the values of columns.
        set -- $LINE
done < inputfile

IFS="$OLDIFS"

You can also use IFS="," read -a arrayname to read elements into an array if you have BASH.

If your columns are separated by ordinary whitespace, leave off the IFS="" part of course.

Assuming nothing more than that the data is a valid text file, while the following traps may not trigger often, I think it's a good idea to mention them for the benefit of novices considering adding these techniques to their repertoire.

Anything that looks like a shell filename pattern (and the possibilities of encountering one increase with increasing shell capability) could modify the data. I would suggest protecting against the worst by using set to toggle globbing.


Whether reading into an array or a string variable, without -r, backslashes and newlines in the data will be lost and a valid text file could be rendered invalid (if its concluding newline is consumed by read).

Regards,
Alister

@mjf... i have quick question i kept the command u send me in a function i want it to return a value....

Code:

 
comm -23 file1.txt defaultfile.txt | xargs -Ifn echo fn "not authenticated" | RCODE = 5

name5 not authenticated

and then do return $RCODE and will use this RCODE in another if statement. ....my doubt is that can i do this change and will it work...


Thanks again for your reply

draghun9,
How you have coded the RCODE above will not work (try it). If you want to code this into a function, then why try to do in all one command line? Use an If/else condition to echo the return code based on results of each compare.

@MJF...
Yeah, i tried and returncode doesnt work....and the thing is even if one line is not authenticated, it should return the value with rcode=5.... as i said i am new to the scripting i am really struck at this part (no clue abt how to proceed with if/else for this requirement)...please help.