Shell Programming and Scripting

Let me give you a complete example what I am trying to achieve.

1. Below is the log file structure where I need 2,5 and 14th column of the logs after grepping through the linkId=1ddoic.

Log file structure:-

abc.com 20120829001415 127.0.0.1 app none11111 sas 0 0 N clk Mozilla/5.0 id=82c6a15ca06b2372c3b3ec2133fc8b14 referrer=google.com linkId=1ddoic

abc.com 20120829001416 127.0.0.1 dyn UD3BSAp8appncXlZ UD3BSAp8app4xHbz 0 0 N page Mozilla/5.0 id=82c6a15ca06b2372c3b3ec2133fc8b14 segments=

2. Now for the 1st log you can see I have invalid(none11111) 5th column. So I have to look for the actual 5th column value. 'id' column will help me to find that. So you have to run another grep based on the 'id' value so that you can find the actual 5th column in the same log file.
3. If you see the second log it has the exact matching 'id' value. So what I have to do I have to take the 5th column(UD3BSAp8appncXlZ) from the second log instead of the invalid one(none11111).
4. So basically I have to run another command which will do that. This is a conditional command which only runs when there is invalid 5th column.

Output:-

20120829001415, UD3BSAp8appncXlZ, linkId=1ddoic

Note:- I have bunch of log files where I have to perform the above procedure. But I have to come up with a single file as output after grepping through all the log files.
It has a format like abc-2012-10-01_00000,abc-2012-10-01_00001.... etc.

Thanks in advance.

grep is not conditional. It matches lines. It's not a language...

awk is a language, and can do things conditionally. You can even tell it to read the same file twice... On the first pass, save the ID's. On the second pass, fill in missing ones...

Code:

awk 'NR==FNR && !($5 ~ /none/) {
        A[$12]=$5 # Save ID and key for later
        next # Skip to next line
}

($5 ~ /none/) && ($12 in A) { $5=A[$12] } 1' logfile logfile > output

Hi Corona,

Thank you for your reply.
I am a newbie in unix. It would be very helpful if you explain your code little bit. I did not understand how do I run the loop through all the log files.

Thanks.

awk is a language which loops over all lines by default.

So, for every single line, it runs both code blocks.

Code:

awk '# NR is the total line number.
# FNR is the line number in the file.
# When they are the same, it must be in the right file.
#
# $5 is the 5th field.  When it does not contain none,
# it is considered to be a valid line, and the 12th column
# and 5th column get saved for later in the array A.
# Then it skips to the next line without executing the block below.
NR==FNR && !($5 ~ /none/) {
        A[$12]=$5 # Save ID and key for later
        next # Skip to next line
}

# When the 5th column contains none, try to look
# it up in A.  If found, set the 5th column to its contents.
# The 1 at the end of the line tells it to print every line.
($5 ~ /none/) && ($12 in A) { $5=A[$12] } 1' logfile logfile

We give awk logfile twice, so it reads the same file twice. The first time it saves names for lookup. The second time, it tries to look up entries and substitute the data you wanted before printing every line.