Simple awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Simple awk
# 1  
Old 07-28-2013
Question Simple awk

Hi, this must be a simple but this is my first interaction with shell and awk.

following is a log file needed to parse (2 lines separated by a line break for clarity):
Code:
2013-07-27 13:32:09,043 - ERROR - PerformanceUtility - Thread-14 - Performance - 9b348407-4f57-4983-a057-a55669821f68 | BidirectionalCompoundProviderMediation | ProviderAdapter.Response | B1.RPBLBKL04.EGS107 | RetryID: 1374949928859 -> 1374949928859 [ 0 ] | EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ] | StateLoading: 1374949928870 -> 1374949928871 [ 1 ] | CorrelID: 1374949928859 -> 1374949928871 [ 12 ] | TotalID: 1374949928859 -> 1374949928871 [ 12 ] | com.bofa.rp.engine.v004.broker.optimus.task.OptimusResponseTask TaskExecution: 15 | com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0 | Propogate: 1374949928887 -> 1374949928945 [ 58 ] | Delivery: 1374949928887 -> 1374949928945 [ 58 ] | Orchestration: 1374949928871 -> 1374949928945 [ 74 ] | UpdateEngineRecord: 1374949928945 -> 1374949928948 [ 3 ] | Propogate: 1374949928950 -> 1374949929041 [ 91 ] | PublishAudit: 1374949928948 -> 1374949929042 [ 94 ] | Propogate: 1374949929043 -> 1374949929043 [ 0 ] | PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ] | Execution: 1374949928859 -> 1374949929043 [ 184 ]

2013-07-27 13:32:08,723 - ERROR - PerformanceUtility - Thread-13 - Performance - b089c04a-a936-45f8-a94b-196da5494424 | BidirectionalCompoundProviderMediation | ProviderAdapter.Request | B1.RPBLBKL04.EGS105 | RetryID: 1374949928412 -> 1374949928412 [ 0 ] | EngineRecordLoad: 1374949928412 -> 1374949928419 [ 7 ] | StateLoading: 1374949928419 -> 1374949928419 [ 0 ] | CorrelID: 1374949928412 -> 1374949928419 [ 7 ] | TotalID: 1374949928412 -> 1374949928419 [ 7 ] | com.bofa.rp.engine.v004.broker.optimus.task.OptimusResponseTask TaskExecution: 14 | com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0 | Propogate: 1374949928434 -> 1374949928612 [ 178 ] | Delivery: 1374949928433 -> 1374949928612 [ 179 ] | Orchestration: 1374949928419 -> 1374949928612 [ 193 ] | UpdateEngineRecord: 1374949928612 -> 1374949928616 [ 4 ] | Propogate: 1374949928618 -> 1374949928620 [ 2 ] | PublishAudit: 1374949928616 -> 1374949928620 [ 4 ] | Propogate: 1374949928620 -> 1374949928723 [ 103 ] | PublishRecovery: 1374949928620 -> 1374949928723 [ 103 ] | Execution: 1374949928412 -> 1374949928723 [ 311 ]

This is my attempt ...

Code:
#!/bin/awk
FILENAME="EnginePerformance.log"
cat $FILENAME | grep -E "(RPALBKL|RPBLBKL|RPCLBKL)" | awk -F '|'
'{ 
       if ($3 ~ /.*Request.*/)
          print $3 $4 $11 $12 $22 $23 $24 $25 $26 $27         
       else if ($3 ~ /Response/) {
          print $3 $4 $6 $19 $20 $21 $22 $23 
       }
}'

Thanks for your help!
# 2  
Old 07-28-2013
Hi liv2luv,

check this out:

Code:
#!/bin/bash

or
Code:
 #!/bin/ksh

depending on the type of sheel ,you should keep at the begining. and not #!/bin/awk

Code:
$ awk -F '|' '/RPALBKL/||/RPBLBKL/||/RPCLBKL/{if($3~/\.Request/)print $3 $4 $11 $12 $22 $23 $24 $25 $26 $27;else if($3~/\.Response/)print $3 $4 $6 $19 $20 $21 $22 $23 } ' file
 ProviderAdapter.Response  B1.RPBLBKL04.EGS107  EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ]  PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ]  Execution: 1374949928859 -> 1374949929043 [ 184 ] 2013-07-27 13:32:08,723 - ERROR - PerformanceUtility - Thread-13 - Performance - b089c04a-a936-45f8-a94b-196da5494424  BidirectionalCompoundProviderMediation  ProviderAdapter.Request  B1.RPBLBKL04.EGS105

---------- Post updated at 02:38 AM ---------- Previous update was at 02:35 AM ----------

Adding to that , you can get similar type of output : by adding "|"with the print statement as your output is pipe delimited:

print $3,"|", $4 ,"|", $11,"|", ....
# 3  
Old 07-28-2013
Many thanks, the code worked just perfect!

Hopefully I will be able to learn this well one day..
# 4  
Old 07-28-2013
Here is a very slightly more complex awk script that should be much more efficient than what you were doing.
Code:
#!/bin/ksh
FILENAME=EnginePerformance.log
awk 'BEGIN { FS = " [|] "; OFS = " | " }
! /RPALBKL|RPBLBKL|RPCLBKL/ { next }
$3 ~ /Request/  { print $3,$4,$11,$12,$22,$23,$24,$25,$26,$27 }
$3 ~ /Response/ { print $3,$4,$6,$19,$20,$21,$22,$23 }' "$FILENAME"

I used ksh, but this script will work with any shell that recognizes basic Bourne shell syntax. If you are using a Solaris/SunOS system, use /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk instead of just awk. (This is also true if you use the awk scripts suggested by rveri.)

The first line in the awk script above sets the input and output field separators to " | ". The second line replaces your grep. And the third and fourth lines replace your if statements.
Note that your sample input only had 20 fields, but you're printing the contents of fields up to field #27. That is why there are empty fields at the end of the output lines this script produces.

Note also that your sample input had lines that were about 1150 bytes long. If your real input has lines that are longer than LINE_MAX on your system (you can get the limit on your system using the command getconf LINE_MAX), the behavior of awk and grep is unspecified. The standards specify LINE_MAX must be at least 2048; and 2048 is the limit on many systems. If you can't use awk and grep to get reliable results on your system, the solution to your problem will be much more complex.

You didn't show us what output you wanted, so rveri and I made similar guesses at what you wanted. The difference between rveri's scripts and mine is that I assume your field separator (for input and output) was <space><vertical bar><space> where rveri removed the <vertical bar> in the first suggestion and added more spaces in the output with the suggested edit.

With the sample input given in the 1st message in this thread, the output my suggestion above produces is:
Code:
ProviderAdapter.Response | B1.RPBLBKL04.EGS107 | EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ] | PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ] | Execution: 1374949928859 -> 1374949929043 [ 184 ] |  |  | 
ProviderAdapter.Request | B1.RPBLBKL04.EGS105 | com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0 | Propogate: 1374949928434 -> 1374949928612 [ 178 ] |  |  |  |  |  |

The original code posted by rveri produces:
Code:
 ProviderAdapter.Response  B1.RPBLBKL04.EGS107  EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ]  PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ]  Execution: 1374949928859 -> 1374949929043 [ 184 ]
 ProviderAdapter.Request  B1.RPBLBKL04.EGS105  com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0  Propogate: 1374949928434 -> 1374949928612 [ 178 ]

Note the leading spaces and the lack of any indication at the end of the output lines corresponding to the empty output fields.

With the suggested edit to rveri's code, it produces:
Code:
 ProviderAdapter.Response  |  B1.RPBLBKL04.EGS107  |  EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ]  |  PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ]  |  Execution: 1374949928859 -> 1374949929043 [ 184 ] |  |  | 
 ProviderAdapter.Request  |  B1.RPBLBKL04.EGS105  |  com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0  |  Propogate: 1374949928434 -> 1374949928612 [ 178 ]  |  |  |  |  |  |

Note the leading spaces and the added spaces separating existing input fields that are copied to the output. This presents the empty fields at the end of the line the same way my suggestion does it.

Maybe one of these three will be close to what you want.
# 5  
Old 07-29-2013
Thanks Don, honestly, I can not appreciate the improvement in code given my limitation in understanding it.

The code given by rveri which produces space separated values (or rather just the tokens with out pipe) does suffice. Thanks again for your help!

---------- Post updated at 05:04 PM ---------- Previous update was at 11:22 AM ----------

The servers are cloned from a single image, however on some of the servers, I am gettign this:
Code:
./show-log.sh[8]: : cannot execute [Is a directory]

What could be the reason. Shell is ksh. Google search did not return any satisfactory responses.
# 6  
Old 07-29-2013
Seeing the contents of show-log.sh would be helpful in deducing why show-log.sh is misbehaving.
# 7  
Old 07-29-2013
Code:
$ cat show-log.sh
#!/bin/ksh
## This script is to view the wanted values in EnginePerformance Log for both Request and Response logs from message broker.
FILENAME="EnginePerformance.log"

## Check if the file exist
if [ -e $FILENAME ];
  then ""
else
  echo "$FILENAME does not exist!"
  exit 1
fi

## Tail the file and parse request and response logs
tail -1000f $FILENAME | awk -F '|' '/RPALBKL/||/RPBLBKL/||/RPCLBKL/{
if($3~/\.Request/)
   print $3 $4 $11 $12 $22 $23 $24 $25 $26 $27;
else if($3~/\.Response/)
   print $3 $4 $6 $19 $20 $21 $22 $23 }
'
exit 0

The code is same across all servers. Also the servers are cloned from a single image. It is indeed surprising to see the in-consistent behavior. Any hint will be helpful. Thanks.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Simple awk help

Hey all, so I'm using AWK in a project at work, to generate xml from csv. So far things are going relatively smoothly, but I have one thing I can't figure out. For every field in each row, I must generate <entry name=KWNamex>Field</entry> Then I will need to pull data from a second file... (6 Replies)
Discussion started by: Parrakarry
6 Replies

2. Shell Programming and Scripting

Simple awk use

Hi, I have a file such that: 40454,31,48,4,1304.967741935484,1 31708,25,48,4,1268.32,1 20900,64501,671,788,0.3240259840932699,0 20137,51358,834,743,0.3920908135051988,0 I want to replace the 6th column by "ones" if it is 1, and with "zeros" if it is 0. Thanks. (6 Replies)
Discussion started by: shekhar2010us
6 Replies

3. Shell Programming and Scripting

Need help with simple comparison in AWK

Hi, I'm new to AWK and I'm having problems comparing a field to a string variable. /ARTIST/ {x = $2} $1 ~ x {print $0}My code tries to find a record with the string "ARTIST". Once it finds it, it stores the second field of the record into a variable. I don't know what the problem is for the... (7 Replies)
Discussion started by: klusps
7 Replies

4. Shell Programming and Scripting

another simple awk problem

Hello; I need to print two previous lines after searching for a reg exp: awk '/haywood/' should produce the following =================== p9J46THe020804 89922 Tue Oct 18 21:06 MAILER-DAEMON (host map: lookup (haywood.com): deferred) ... (1 Reply)
Discussion started by: delphys
1 Replies

5. Shell Programming and Scripting

Simple awk problem II

Hello; Trying to figure out how to keep just the contents between the two search lines: awk '/regexp_1/ ,/regexp_2/' I do not want lines containing regexp_1 and regexp_2 in the output. Thank you for any ideas Video tutorial on how to use code tags in The UNIX and Linux Forums. (5 Replies)
Discussion started by: delphys
5 Replies

6. UNIX for Dummies Questions & Answers

Please explain this simple AWK example

awk '!_++' Most importantly, I want to know what the underscore does "!_" But ideally, please breakdown the whole thing. It is supposed to remove duplicate lines when found in a file. (1 Reply)
Discussion started by: glev2005
1 Replies

7. Shell Programming and Scripting

help with simple awk script

Hi, I just don't understand awk. I think I'm close here but can someone give me a hand to finish this little parsing routine ? the input file is formatted like this: District 2502110 Gsub 2384889 Gsub 1428180 District 2502220 Gsub 1466390 Gsub 1466389 Gsub 1466388 Gsub 1466386 Gsub... (4 Replies)
Discussion started by: fwellers
4 Replies

8. Shell Programming and Scripting

simple awk help

Whats the syntax to find all lines that matches a text and print out specific fields after the match? ex: 1: some random text 2: Full name: John E. Smith 3: some random text 4: Full name: Mary J. Lue 5: some random text So I'd like to print out First names or last names or everything... (2 Replies)
Discussion started by: linuxdude
2 Replies

9. Shell Programming and Scripting

simple awk

when I execute this awk stmt .. awk "/log_directory/ { print $5}" /opt/dba/oraadmin/tools/tmp_purge_op.log it's returning the whole line as .. IRMD118_LISTENER1 parameter "log_directory" set to /opt/oracle/10.2/network/log/ my expected output is : /opt/oracle/10.2/network/log what... (7 Replies)
Discussion started by: talashil
7 Replies

10. Shell Programming and Scripting

Help with a simple script using awk

I need a hand with this simple script, in Unix i have a variable called portal: $ echo $portal chu0 when i use awk this variable is not recognized. How can i make awk recognize and print the value of this variable, since the output is not shown i.e. awk ' BEGIN {printf("%4s\t%14s\n",... (3 Replies)
Discussion started by: alexcol
3 Replies
Login or Register to Ask a Question