Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Awk help needed

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Awk help needed
# 1  
Old 10-27-2009
Awk help needed
I have a log file monitor script that checks through a log file for a string. I use awk to search the log file, starting at the last checked line, for the specified string and then output the count and the last row number checked. The part of the script that does all the work is here:

Code:
set -- $(awk -v lines=$lineslastrun -v pattern="$searchpattern" '
  BEGIN { count=0 }
  NR > lines && $0 ~ pattern { count++ }
  END { print NR; print count }
' $logfile)
totallines=$1
matches=$2

This particular log is a java app server log and we're looking for the occurance of "Full GC". Now, they want to only count the Full GC if the amount of recovered space is less than 100000. This is what the total output of a Full GC looks like in the log:

Code:
844174.631: [Full GC 844174.631: [ParNew
Desired survivor size 13402112 bytes, new threshold 15 (max 15)
- age   1:    1365536 bytes,    1365536 total
- age   2:     122888 bytes,    1488424 total
- age   3:     225712 bytes,    1714136 total
- age   4:     320592 bytes,    2034728 total
- age   5:       8368 bytes,    2043096 total
- age   6:     483952 bytes,    2527048 total
- age   7:      28256 bytes,    2555304 total
- age   8:      63768 bytes,    2619072 total
- age   9:     106576 bytes,    2725648 total
- age  10:      11960 bytes,    2737608 total
- age  11:        344 bytes,    2737952 total
- age  12:     111824 bytes,    2849776 total
- age  13:     105072 bytes,    2954848 total
- age  14:     189000 bytes,    3143848 total
- age  15:       1544 bytes,    3145392 total
: 212035K->3096K(235968K), 0.0180000 secs] 2079826K->1871065K(2333120K), 0.0181290 secs]

I have bolded the 2 numbers I need to calculate. If the first number minus the second number is less than 100000 then I want to advance the count.

I'm stuck. I've created a mock log and messed around a bit and only came up with this:

Code:
awk '/Full GC/,/^:/ {if (/^:/) print $5}' testawk.data | awk -F \( '{ print $1 }' | tr -d '\-\>' | tr 'K' ' ' | awk '{if ($1-$2<100000) print}'

I can figure out how to get that into my set command above and get it to output the count. I can't figure out how I'll output the last line number of the log that was checked. I'm hoping someone can take the time to show me how to do this properly.

Thanks
# 2  
Old 10-27-2009
start with this:
Code:
echo ': 212035K->3096K(235968K), 0.0180000 secs] 200000K->90K(2333120K), 0.0181290 secs]' | nawk '{n=split($5, a, "[->(]"); print (a[1]-a[3]<100000)? "less" : "NOTless"}'

# 3  
Old 10-28-2009
Thanks for the nudge in the right direction. First off, I didn't know I could perform the calculation without removing the "K" from each number. That certainly saved some steps. Besides that, your rewrite allows me to fit it all in a single awk which makes outputting the number of lines easy. Here's what I came up with:

Code:
awk 'BEGIN { count=0 } NR > 10 && /Full GC/,/^:/ {if (/^:/) split($5, a, "[->(]"); else next; if (a[1]-a[3]<100000) count++} END { print NR; print count }' testawk.data

The 'NR > 10' will actually be 'NR > lines' once I incorporate it into the script. You can see it in my first post. This allows me to skip over log file entries checked in previous runs of the script.

Thanks again

---------- Post updated 10-28-09 at 04:07 PM ---------- Previous update was 10-27-09 at 06:25 PM ----------

All is working well except I can't seem to pass the search string to awk. Here's the snippit:

Code:
set -- $(awk -v lines=$lineslastrun -v pattern="$searchpattern" '
  BEGIN { count=0 }
  NR > lines && /Full GC/,/^:/ {if (/^:/) split($5, a, "[->(]"); else next; if (a[1]-a[3]<100000) count++}
  END { print NR; print count }
' $logfile)
totallines=$1
matches=$2

I would like to change "Full GC" to "pattern" so I can pass the string as a variable to awk. Unfortunately it only works if I put in as it is above. I have tried many versions of single and double quotes:

Code:
NR > lines && /pattern/,/^:/ {if (/^:/) split($5, a, "[->(]"); else next; if (a[1]-a[3]<100000) count++}
NR > lines && "/pattern/",/^:/ {if (/^:/) split($5, a, "[->(]"); else next; if (a[1]-a[3]<100000) count++}
NR > lines && /"pattern"/,/^:/ {if (/^:/) split($5, a, "[->(]"); else next; if (a[1]-a[3]<100000) count++}

Plus many more. None worked. Any suggestions?
# 4  
Old 10-29-2009
Code:
$0 ~ pattern

# 5  
Old 10-29-2009
That's what I had before when I was just looking for 1 line. Now I'm looking for a block of text.
# 6  
Old 10-29-2009
Code:
NR > lines && $0 ~ pattern,/^:/

# 7  
Old 10-29-2009
Ok. That did it. Working perfectly. Thank you.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk - help needed please

Hi.. have a file as below, appreciate if someone can help on this 143|500| 10| 23353 22131 23355 23354 23358 23352 23357 23350 23349 23351| RAID5 213|1008| 9| 22419 22412 221 22413 22414 22416 22417 22415 22418| RAID6 1088|500| 5| 22243 22240 22244 22242 22241| RAID5 322|1200| 12|... (6 Replies)
Discussion started by: richard0@rediff
6 Replies

2. Shell Programming and Scripting

awk help needed

Hi Everyone, i have following in my file 1 2 3 4 5 6 . . 100 and now i want the output as 1 4 7 ..........so on..............97 100 (10 Replies)
Discussion started by: zozoo
10 Replies

3. Shell Programming and Scripting

awk help needed

Hi Experts, I have a file (file 1) with several columns and I need to create 2 files based on the data of 20th column of file 1. Criteria 1 : If the 20th field of file1 is empty , copy the entire records to file 2. I am successfully able to do this with the following awk code : awk... (2 Replies)
Discussion started by: nua7
2 Replies

4. Shell Programming and Scripting

awk help needed

Hi, i have input records as shown below. 4097,Probe3,G10,255,05/17/2011 12:44:03:185,NULL,05/17/2011 12:39:03:180,05/17/2011... (1 Reply)
Discussion started by: raghavendra.nsn
1 Replies

5. UNIX for Dummies Questions & Answers

help needed for awk

Dear all, I am new to use unix. I run the following command and got the error. Anyone knows how should I modify the command. Thanks a lot! $ for chr in 'seq 1 23'; do awk 'BEGIN {print "T","pheno";}{print "M",$2}' out_${chr}.map > dat_${chr}.dat; done error message:... (2 Replies)
Discussion started by: forevertl
2 Replies

6. Shell Programming and Scripting

Awk Help needed

hi, I have input file woth records as shown below OCSMRC_OK,7057348733,+0.00,0,18/05/2010 23:42:19,BellMobility,302610000918553,0006056099,B30,686505,686505,OCS_MRC,+49.14,0,0 ,0,0, OCSPPKB_NOK,4163460120,+1.25,0,18/05/2010... (4 Replies)
Discussion started by: raghavendra.cse
4 Replies

7. Shell Programming and Scripting

help needed in awk

Hi , i have a file a.txt like this: far near veryfar toonear typeset var1=veryfar to extract the text between two strings i use the following command : awk '/far/,$veryfar/' a.txt its not working can nyone tell pls whats wrong in it ? i doubt can we use variable in awk like this... (3 Replies)
Discussion started by: santosh1234
3 Replies

8. Shell Programming and Scripting

awk help needed

How do I alter this command so that it prints only the second comma delimited field from line number 3? Secondly, how do you redirect the output to a variable called TEST? Thanks (cat BATCH007.TXT | awk 'BEGIN { FS = "," } ; {print $2 }') (5 Replies)
Discussion started by: ddurden7
5 Replies

9. Shell Programming and Scripting

awk help needed

I am trying to write a script that will parse out the e-mail address of a person from the name of a file in a directory. Example: filename is: /home/myname/first.middle.last@email.com.xls I want to extract just the email address and mail the file to that address. I want to send the... (6 Replies)
Discussion started by: Drenhead
6 Replies

10. Shell Programming and Scripting

awk help needed.

Dear All, I want to compare column 3 and column 4 of files(File1 and File2) and want to get the matched records in one file and rejected records in another file.I have tried an awk script,but as Iam not good in it,I cant able to continue.Can anyone help me in this one. File 1... (0 Replies)
Discussion started by: cskumar
0 Replies
Login or Register to Ask a Question