Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk to find lines containing word that occur multiple times

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk to find lines containing word that occur multiple times
# 1  
Old 02-20-2013
awk to find lines containing word that occur multiple times
i have a script that scans a log file every 10 minutes. this script remembers the last line of the log and then uses it to continue monitoring the log when it runs again 10 minutes later.

the script searches the log for a string called MaxClients.

now, how can i make it so that when the script runs, it'll let me know when it finds not just 1 but 3 lines containing MaxClients?

Code:
awk 'FNR=>'5034' logfile | egrep MaxClients

sounds simple enough. but there's a catch. lets say the log file can contains:

Code:
Warning MaxClient is having issues - please
MaxClients java error...yams potato turkey
MaxClients error Fatal exception known known
MaxClients error Fatal exception known known
Could not complete. Error found. MaxClient initiated
MaxClients error Fatal exception known known
Could not complete. Error found. MaxClient initiated

in the above example, the script should only output the lines:

Code:
MaxClients error Fatal exception known known
MaxClients error Fatal exception known known
MaxClients error Fatal exception known known

This is because, this is the line that occurred at least 3 times when the script scanned the log at its 10 minute interval.
# 2  
Old 02-20-2013
Code:
grep "^MaxClients error Fatal exception known known" file

This should do! Smilie
This User Gave Thanks to PikK45 For This Post:
SkySmart
# 3  
Old 02-20-2013
A pointer (may be it's 90% of the solution!):
Code:
awk '/MaxClients/{a[$0]++}END{for(i in a) if(a[i]>=3) for(j=1;j<=a[i];j++) print i}' file

This User Gave Thanks to elixir_sinari For This Post:
SkySmart
# 4  
Old 02-20-2013
First of all no need to pipe awk output to egrep. You can save that pipeline because awk is capable of doing what egrep can.

Will this code work for you?
Code:
awk 'NR>=5034&&/^MaxClients error/{++c}c==3{o=$0 RS $0 RS $0; print o; exit 1}' logfile

This User Gave Thanks to Yoda For This Post:
SkySmart
# 5  
Old 02-20-2013
Quote:
Originally Posted by bipinajith
First of all no need to pipe awk output to egrep. You can save that pipeline because awk is capable of doing what egrep can.

Will this code work for you?
Code:
awk 'NR>=5034&&/^MaxClients error/{++c}c==3{o=$0 RS $0 RS $0; print o; exit 1}' logfile

this looks like it'll work, but what happens if there are more than 3 occurrences? i think this code will only show the 3 lines. however, what happens if there are more than 3 lines?

i should have been clearer in my post. sorry for that.
# 6  
Old 02-20-2013
If you want to print each time the number of occurrences reaches 3, then remove the exit 1 statement and reset counter c=0
Code:
awk 'NR>=5034&&/^MaxClients error/{++c}c==3{o=$0 RS $0 RS $0; print o; c=0}' logfile

Now the code will print every times 3 occurrences are found. If this is not you want, modify as per your requirement. I hope this helps.
This User Gave Thanks to Yoda For This Post:
SkySmart
# 7  
Old 02-20-2013
Quote:
Originally Posted by bipinajith
If you want to print each time the number of occurrences reaches 3, then remove the exit 1 statement and reset counter c=0
Code:
awk 'NR>=5034&&/^MaxClients error/{++c}c==3{o=$0 RS $0 RS $0; print o; c=0}' logfile

Now the code will print every times 3 occurrences are found. If this is not you want, modify as per your requirement. I hope this helps.
Thank you. one last question. is the below the best way to include search strings i want to exclude?

Code:
awk 'NR>=1&&/MaxClients/ && !/java|could not.*problem found|panic() failure seen|aborting [ERROR]/ {++c}c==3{o=$0 RS $0 RS $0; print o; c=0}' logfile

in the above, i'm basically saying i want to ignore all lines that contain MaxClients that also contain any of these
Code:
"java|could not.*problem found|panic() failure seen|aborting [ERROR]".

Last edited by SkySmart; 02-20-2013 at 01:43 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to find=grep or maybe sed/awk for multiple lines of text?

Hi, I am running the following: PASS="username/password" sqlplus -s << EOF | grep -v "^$" $PASS set feedback off set heading off set termout off select name from v\$database ; exit EOF Which gives ERROR: ORA-28002: the password will expire within 5 days PSMP1 (1 Reply)
Discussion started by: newbie_01
1 Replies

2. Shell Programming and Scripting

Find word in a line and output in which line the word occurs / no. of times it occurred

I have a file: file.txt, which contains the following data in it. This is a file, my name is Karl, what is this process, karl is karl junior, file is a test file, file's name is file.txt My name is not Karl, my name is Karl Joey What is your name? Do you know your name and... (3 Replies)
Discussion started by: anuragpgtgerman
3 Replies

3. Shell Programming and Scripting

Shell Script @ Find a key word and If the key word matches then replace next 7 lines only

Hi All, I have a XML file which is looks like as below. <<please see the attachment >> <?xml version="1.0" encoding="UTF-8"?> <esites> <esite> <name>XXX.com</name> <storeId>10001</storeId> <module> ... (4 Replies)
Discussion started by: Rajeev_hbk
4 Replies

4. Shell Programming and Scripting

Count same word which has come many times in single lines & pars

can i get a simple script for , Count same word which has come many times in single lines & pars Eg file would be == "Thanks heman thanks thanks Thanks heman thanks man" So resullt should be Thanks = 5 heman=2 man = 1 thanks in advance :) Please use code tags for code and... (1 Reply)
Discussion started by: heman96
1 Replies

5. UNIX for Dummies Questions & Answers

Extracting data between specific lines, multiple times

I need help extracting specific lines in a text file. The file looks like this: POSITION TOTAL-FORCE (eV/Angst) ----------------------------------------------------------------------------------- 1.86126 1.86973 1.86972 ... (14 Replies)
Discussion started by: captainalright
14 Replies

6. Shell Programming and Scripting

Sed or Awk for lines between two strings multiple times and keep the last one

Hi, I am trying to get lines between the last occurrences of two patterns. I have files that have several occurrences of “Standard” and “Visual”. I will like to get the lines between “Standard” and “Visual” but I only want to retain only the last one e.g. Standard Some words Some words Some... (4 Replies)
Discussion started by: damanidada
4 Replies

7. UNIX for Dummies Questions & Answers

Find string multiple times, same line

Hi everybody, Fairly simple question here: I need an awk, sed, or grep command that will find the same string multiple times on one line needs to return all lines which contain .02 twice. I do know the exact number of characters in between the two occurrences of .02 if that helps, all... (7 Replies)
Discussion started by: jgrosecl
7 Replies

8. Shell Programming and Scripting

AWK Duplicate lines multiple times based on a calculated value

Hi, I'm trying to create an XML sitemap of our dynamic ecommerce sites SEO Friendly URLs and am trying to create the initial page listing. I have a CSV file that looks like the following and need duplicate the lines based on a value which needs calculating. ... (2 Replies)
Discussion started by: jamesfx
2 Replies

9. Shell Programming and Scripting

how many times a word occur in afile

i want a shell script program for how many times a word occur in a file. i need not the line number but i want the counts of the particular word for eg:- hai how r u.. i am from andhra pradesh.. i am from tenali.i need this answer.i need it urgently.. i hope u will answer this ... ... (9 Replies)
Discussion started by: madhu.it
9 Replies

10. Shell Programming and Scripting

TO find the word which occurs maximum number of times

Hi Folks !!!!!!!!!!!!!!!!!!! My Requirement is............. i have a input file: 501,501.chan 502,502.anand 503,503.biji 504,504.raja 505,505.chan 506,506.anand 507,507.chan and my o/p should be chan->3 i.e. the word which occurs maximum number of times in a file should be... (5 Replies)
Discussion started by: aajan
5 Replies
Login or Register to Ask a Question