pattern search for multiple log files and counting


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting pattern search for multiple log files and counting
# 1  
Old 09-10-2009
pattern search for multiple log files and counting

I have 10 appservers and each appserver has 4 jvms . Each of these logs is archived and stored on a nfs directory . For example the files are

/logs/200907/ap1-jvm1.server.log.20090715.gz
/logs/200907/ap2-jvm2.server.log.20090714.gz
/logs/200908/ap1-jvm1.server.log.20090812.gz

I want to search for say some pattern from these files from say august 15th to sept 15th and give me a count of that pattern .For example I want a daily count from aug15-sept 15 for app server and jvm for pattern "Exception" from the log . It should look like

-----
Aug 15
3434
Aug 16
2323
....
....
Sept10
5656

thanks
Gubbs
# 2  
Old 09-10-2009
It is very simple, if i got your question correctly.,

Code:
$ zgrep -c 'Exception' FILENAMEs

But it will be good to ask a question, along with what you tried ? and what is the issue you are facing ?
# 3  
Old 09-10-2009
Thanks for the response . Here is what is working if I need it for one day

Code:
for m in 25;do for q in 1 2 3  ; do for r in 1 2  ; do ssh -q hostname "/bin/zcat /logs/ap${q}-jvm${r}.server.log.200908${k}.gz|grep 'Exception'|wc -l";done;done;done|awk '{SUM+=$1} END {print SUM}'

This gives me for that particular day ...I am having trouble when I want to give it a range .Essentially "wc -l" does the same as "zgrep -c "

For example I tried

Code:
for m in 20 21 22 23 24 25;do for q in 1 2 3 4 5 6 7 8 9 10 ; do for r in 1 2 3 4 ; do ssh -q hostname "/bin/zcat /logs/ap${q}-jvm${r}.server.log.200908${k}.gz|grep 'Exception'|wc -l|awk '{SUM+=$1} END {print SUM}'";done;done;done

I get the following error
Code:
awk: cmd. line:1: {SUM+=} END {print SUM}
awk: cmd. line:1:       ^ syntax error


so basically I want a count of "Exception" per day from multiple files and want to report it for each day

Code:
day1 
23232
day 2
23232
day3 
42323


Last edited by vgersh99; 09-10-2009 at 05:57 PM.. Reason: code tags, PLEASE!
# 4  
Old 09-10-2009
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Pattern search multiple files

#!/usr/bin/ksh a="Run successfully" cd $APPS ls -l *.txt | while read $txt do if then cp $APPS/$txt cp $hist/$txt else rm $APPS/$txt echo "Files has been removed" fi done New in shell script please help me out Around 100 txt files in $APPS dir i want to search pattern from... (8 Replies)
Discussion started by: Kalia
8 Replies

2. Shell Programming and Scripting

Search multiple pattern in a file

I have a sample file with following output: HTTP/1.1 200 OK User: admin Set-Cookie: AMBARISESSIONID=y3v3648yqcno32nq478kw7ar;Path=/;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Content-Type: text/plain Vary: Accept-Encoding, User-Agent Content-Length: 6057 Server:... (4 Replies)
Discussion started by: saurau
4 Replies

3. Linux

Search multiple pattern from list

I am working on AIX operating system. I want to search list of Article Id for given Set Date (which are present in a seperate file input.txt) art_list.csv ------------ "Article ID" |"Ad Description" |"Pyramid"|"Pyramid Desc "|"ProductTypeId"|"Set Date "|... (3 Replies)
Discussion started by: rajivrsk
3 Replies

4. Shell Programming and Scripting

awk with multiple pattern search

Want to fetch a column with multiple pattern using awk. How to achieve the same. Tried cat test address : 10.63.20.92/24 address : 10.64.22.93/24 address : 10.53.40.91/24 cat test | awk '{print $3}' |awk -F "/" '{print $1}' 10.63.20.92 10.64.22.93 10.53.40.91 Is there any... (2 Replies)
Discussion started by: Manasa Pradeep
2 Replies

5. Shell Programming and Scripting

Multiple search pattern

Hello :) I have this file cat employee_list Name : jack Gender: m ID : 4512 DOB : 03/27/1980 hire date : 04/23/2012 Nationality: US marital status : single ===================== Name : mick Gender: m ID : 1256 DOB : 03/27/1970 Hire date : 012/10/2011 Nationality: US Marital... (4 Replies)
Discussion started by: Sara_84
4 Replies

6. Shell Programming and Scripting

Counting occurrences of all words in multiple files

Hey Unix gurus, I would like to count the number occurrences of all the words (regardless of case) across multiple files, preferably outputting them in descending order of occurrence. This is well beyond my paltry shell scripting ability. Researching, I can find many scripts/commands that... (4 Replies)
Discussion started by: twjolson
4 Replies

7. Shell Programming and Scripting

multiple files: counting

In a directory, I have 5000 multiple files that contains around 4000 rows with 10 columns in each file containing a unique string 'AT' located at 4th column. OM 3328 O BT 268 5.800 7.500 4.700 0.000 1.400 OM 3329 O BT 723 8.500 8.900... (7 Replies)
Discussion started by: asanjuan
7 Replies

8. Shell Programming and Scripting

Counting lines in multiple files

Hi, I have couple of .txt files (say 50 files) in a folder. For each file: I need to get the number of lines in each file and then that count -1 (I wanted to exclude the header. Then sum the counts of all files and output the total sum. Is there an efficient way to do this using shell... (7 Replies)
Discussion started by: Lucky Ali
7 Replies

9. UNIX for Dummies Questions & Answers

count the number of files which have a search string, but counting the file only once

I need to count the number of files which have a search string, but counting the file only once if search string is found. eg: File1: Please note that there are 2 occurances of "aaa" aaa bbb ccc aaa File2: Please note that there are 3 occurances of "aaa" aaa bbb ccc... (1 Reply)
Discussion started by: sudheshnaiyer
1 Replies

10. Shell Programming and Scripting

Counting files in a directory that match a pattern

I have 20 files in a direcotry like BARE01_DLY_MKT_YYYYMMDD. The MKT differes for all these files but the remaining syntax remains the same for a particular day. If I am checking for today I need to make sure that there are 20 files that start with BARE01_DLY_MKT_20060720. How can I write a... (31 Replies)
Discussion started by: dsravan
31 Replies
Login or Register to Ask a Question

Featured Tech Videos