Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help to create a script to parse log files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help to create a script to parse log files
# 1  
Old 08-10-2010
Help to create a script to parse log files
Hello everybody,

I need some help here to create a script to parse a log file. Here is a sample of the log file :
Code:
[0720 16:00:00.646257] 0x42258940 (Debug) Cache SUMMARY [s1.meta] attrs now/668 min/668 max/668.
[0720 16:00:00.646262] 0x42258940 (Debug) RSVD SUMMARY [s1.meta] reserved space max requested/128 MB accounted now/0 MB
[0720 16:00:00.646273] 0x42258940 (Debug) VOP SUMMARY [s1.meta] Setattr cnt/14 avg/53+54369 min/13+42 max/334+259525.
[0720 16:00:00.646283] 0x42258940 (Debug) VOP SUMMARY [s1.meta] Open cnt/7 avg/24+187 min/19+175 max/34+231.
[0720 16:00:00.646291] 0x42258940 (Debug) VOP SUMMARY [s1.meta] Close cnt/7 avg/20+37 min/16+29 max/28+54.
[0720 16:00:00.646300] 0x42258940 (Debug) VOP SUMMARY [s1.meta] GetResyncAttr cnt/1 avg/20+25 min/20+25 max/20+25.
[0720 16:00:00.646305] 0x42258940 (Debug) Cache SUMMARY [s3.meta] attrs now/4 min/4 max/4.
[0720 16:00:00.646309] 0x42258940 (Debug) RSVD SUMMARY [s3.meta] reserved space max requested/32 MB accounted now/0 MB
[0720 16:00:00.646320] 0x42258940 (Debug) VOP SUMMARY [s3.meta] Setattr cnt/12 avg/31+63182 min/17+39 max/150+334001.
[0720 16:00:00.646329] 0x42258940 (Debug) VOP SUMMARY [s3.meta] Open cnt/6 avg/22+180 min/18+159 max/31+210.
[0720 16:00:00.646337] 0x42258940 (Debug) VOP SUMMARY [s3.meta] Close cnt/6 avg/19+31 min/17+25 max/29+37.
[0720 16:00:00.646346] 0x42258940 (Debug) VOP SUMMARY [s3.meta] GetResyncAttr cnt/1 avg/19+24 min/19+24 max/19+24.
[0720 16:00:00.646351] 0x42258940 (Debug) Cache SUMMARY [s4.meta] attrs now/387 min/85 max/387.
[0720 16:00:00.646355] 0x42258940 (Debug) RSVD SUMMARY [s4.meta] reserved space max requested/128 MB accounted now/128 MB
[0720 16:00:00.646366] 0x42258940 (Debug) VOP SUMMARY [s4.meta] Setattr cnt/7838 avg/23+41116 min/10+22 max/1515+525846.
[0720 16:00:00.646376] 0x42258940 (Debug) VOP SUMMARY [s4.meta] Open cnt/321 avg/20+35 min/11+15 max/297+343.

What I would like to do in this script is to have two parameters (the log file name and the summary type) as arguments. With these arguments I want it to retrieve the data (summary type, name of the server, action type, value of counter) from the whole file.

For example, if I call it like this : ./parser.sh log1 VOP

I would like to get for example :
VOP SUMMARY s3 GetResyncAttr 1
VOP SUMMARY s3 Open 6
VOP SUMMARY s4 Setattr 7838
VOP SUMMARY s4 Open 321

I hope this is clear enough and that someone will be able to help me as I haven't been able to succed to do that.

Thanks.
# 2  
Old 08-10-2010
Provide the output according to input you posted , because I have not understood why you have not consider S1 and "close" action type.
# 3  
Old 08-10-2010
Quote:
Originally Posted by pravin27
Provide the output according to input you posted , because I have not understood why you have not consider S1 and "close" action type.
Hi pravin,

Ok, let's say I have a log file called log1 with this inside :
Code:
[0720 16:00:00.646257] 0x42258940 (Debug) Cache SUMMARY [s1.meta] attrs now/668 min/668 max/668.
[0720 16:00:00.646262] 0x42258940 (Debug) RSVD SUMMARY [s1.meta] reserved space max requested/128 MB accounted now/0 MB
[0720 16:00:00.646273] 0x42258940 (Debug) VOP SUMMARY [s1.meta] Setattr cnt/14 avg/53+54369 min/13+42 max/334+259525.
[0720 16:00:00.646283] 0x42258940 (Debug) VOP SUMMARY [s1.meta] Open cnt/7 avg/24+187 min/19+175 max/34+231.
[0720 16:00:00.646291] 0x42258940 (Debug) VOP SUMMARY [s1.meta] Close cnt/7 avg/20+37 min/16+29 max/28+54.
[0720 16:00:00.646300] 0x42258940 (Debug) VOP SUMMARY [s1.meta] GetResyncAttr cnt/1 avg/20+25 min/20+25 max/20+25.
[0720 16:00:00.646305] 0x42258940 (Debug) Cache SUMMARY [s3.meta] attrs now/4 min/4 max/4.
[0720 16:00:00.646309] 0x42258940 (Debug) RSVD SUMMARY [s3.meta] reserved space max requested/32 MB accounted now/0 MB
[0720 16:00:00.646320] 0x42258940 (Debug) VOP SUMMARY [s3.meta] Setattr cnt/12 avg/31+63182 min/17+39 max/150+334001.
[0720 16:00:00.646329] 0x42258940 (Debug) VOP SUMMARY [s3.meta] Open cnt/6 avg/22+180 min/18+159 max/31+210.
[0720 16:00:00.646337] 0x42258940 (Debug) VOP SUMMARY [s3.meta] Close cnt/6 avg/19+31 min/17+25 max/29+37.
[0720 16:00:00.646346] 0x42258940 (Debug) VOP SUMMARY [s3.meta] GetResyncAttr cnt/1 avg/19+24 min/19+24 max/19+24.
[0720 16:00:00.646351] 0x42258940 (Debug) Cache SUMMARY [s4.meta] attrs now/387 min/85 max/387.
[0720 16:00:00.646355] 0x42258940 (Debug) RSVD SUMMARY [s4.meta] reserved space max requested/128 MB accounted now/128 MB
[0720 16:00:00.646366] 0x42258940 (Debug) VOP SUMMARY [s4.meta] Setattr cnt/7838 avg/23+41116 min/10+22 max/1515+525846.
[0720 16:00:00.646376] 0x42258940 (Debug) VOP SUMMARY [s4.meta] Open cnt/321 avg/20+35 min/11+15 max/297+343.

Let's say that I call the script with two arguments (the name of the log file and the type of summary I want) : ./parser.sh log1 VOP

The output I should get is that because I have VOP summary type as an argument :

VOP SUMMARY s1 Setattr 14
VOP SUMMARY s1 Open 7
VOP SUMMARY s1 Close 7
VOP SUMMARY s1 GetResyncAttr 1
VOP SUMMARY s3 Setattr 12
VOP SUMMARY s3 Open 6
VOP SUMMARY s3 Close 6
VOP SUMMARY s3 GetResyncAttr 1
VOP SUMMARY s4 Setattr 7838
VOP SUMMARY s4 Open 321
# 4  
Old 08-10-2010
Hi Try this,

parser.pl
Code:
#!/usr/bin/perl

$filename=shift;
$sum_type=shift;
chomp($sum_type,$filename);

open(FH,"<","$filename") || die "cannot open file \n";

while (<FH>) {
chomp;
if (/\s$sum_type\sSUMMARY\s\[(\w+)\.\w+\]\s(\w+)\scnt\/(\d+)\s/) {
#print $_,"\n";
print "$sum_type SUMMARY $1 $2 $3 \n";
}
}
close(FH);

Code:
perl parser.pl logfilename summary_type

Last edited by pravin27; 08-10-2010 at 11:26 AM..
# 5  
Old 08-10-2010
Quote:
Originally Posted by pravin27
Hi Try this,

parser.pl
Code:
#!/usr/bin/perl

$filename=shift;
$sum_type=shift;
chomp($sum_type,$filename);

open(FH,"<","$filename") || die "cannot open file \n";

while (<FH>) {
chomp;
if (/\s$sum_type\sSUMMARY\s\[(\w+)\.\w+\]\s(\w+)\scnt\/(\d+)\s/) {
#print $_,"\n";
print "$sum_type SUMMARY $1 $2 $3 \n";
}
}
close(FH);

Code:
perl parser.pl logfilename summary_type

Hey Pravin,

I tried your code but I don't get any output when I launch the script.
Does it work for you ?
Another question is it the same code if I want to use this as a bash script ?

Thanks.
# 6  
Old 08-10-2010
Its running fine at my end.

This is perl script. 1) you have to create a file with name "parser.pl"
2) make it executable ( chmod +x parser.pl)
3) run the script , you have to provide log filename with path. In my case log file is in /tmp directory and my log filename is testfile.log
(e.g ./parser.pl /tmp/testfile.log VOIP )
# 7  
Old 08-10-2010
Quote:
Originally Posted by pravin27
Its running fine at my end.

This is perl script. 1) you have to create a file with name "parser.pl"
2) make it executable ( chmod +x parser.pl)
3) run the script , you have to provide log filename with path. In my case log file is in /tmp directory and my log filename is testfile.log
(e.g ./parser.pl /tmp/testfile.log VOIP )
Thanks for the help pravin. i had an error in the script that is why it did not work. I do not want to abuse but can you explain me a little bit what is done in the script because I don't understand everything, especially all those signs you have put in there.

Thanks Smilie.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parse log files

Hi all, We are having a sample log like .... test.log:2015.03.17 06:16:24 >> ABC.generateMethod() MethodAException while processing Request! DataForm: Header --- dtd: template.dtd, titleName: berger, requestId: 1503170032131, documentName: invoice123, hostName: acme.net, userName: userABC... (10 Replies)
Discussion started by: tandrei
10 Replies

2. Shell Programming and Scripting

awk Parse And Create Multiple Files Based on Field Value

Hello: I am working parsing a large input file which will be broken down into multiples based on the second field in the file, in this case: STORE. The idea is to create each file with the corresponding store number, for example: Report_$STORENUM_$DATETIMESTAMP , and obtaining the... (7 Replies)
Discussion started by: ec012
7 Replies

3. Shell Programming and Scripting

Perl script to parse all files in the folder

Hello Smart People! I have a perl script that will import xml data into an access db. I would like to modify it so it will automatcially parse through all xml files in the folder. I swa a post but couldnt get it working. her is what my scrip looks like, i only list the top if you need more... (3 Replies)
Discussion started by: cowboymaverick
3 Replies

4. Shell Programming and Scripting

Parse Log & Create Flow Diagram - Ideas/Tools

Hi, I would like to develop a script which parses the log file and generates a flow diagram ( graphical display ). We have an application, for understanding the sequence of functions call made, we have an debug line at "ENTRY/EXIT" of function. I have a small log parsing script which grep the... (2 Replies)
Discussion started by: ennstate
2 Replies

5. Shell Programming and Scripting

awk script to parse results from TWO files

I am trying to parse two files and get data that does not match in one of the columns ( column 3 in my case ) Data for two files are as follows A.txt ===== abc 10 5 0 1 16 xyz 16 1 1 0 18 efg 30 8 0 2 40 ijk 22 2 0 1 25 B.txt ===== abc... (6 Replies)
Discussion started by: roger67
6 Replies

6. UNIX for Dummies Questions & Answers

How can i parse my Unix log files??

Hello, i would like to parse Unix log files and i would like to use a Unix syslog analyzer. I'm going to use Eucalyptus and i would like to parse its log files. Is there any open source/free syslog parser?? Thanks, in advance! (2 Replies)
Discussion started by: g_p
2 Replies

7. Shell Programming and Scripting

Need a Script - Parse the log backup

Hello all, I need a script to parse the backup logs. I am newbie to scripting, please help. (7 Replies)
Discussion started by: sandeep007
7 Replies

8. Homework & Coursework Questions

shell script that can create, monitor the log files and report the issues for matching pattern

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: Write an automated shell program(s) that can create, monitor the log files and report the issues for matching... (0 Replies)
Discussion started by: itian2010
0 Replies

9. Shell Programming and Scripting

Help w/ script to read file and parse log message

Hi, I am working on the script to parsing the specific message like "aaaa" in multiple log files like N1-***,N2-***,N3-***... The script is to find the list of lof files which contains the message "aaaa" and export the list into excel filE. Can anyone give help? Thanks (2 Replies)
Discussion started by: shyork2001
2 Replies

10. Shell Programming and Scripting

shell scripts that parse log files

hi all ,i would like a shell script that parses log files and checks the contents for any anonalities,please help,thanks (4 Replies)
Discussion started by: trueman82
4 Replies
Login or Register to Ask a Question