Help to create a script to parse log files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help to create a script to parse log files
# 8  
Old 08-10-2010
Please post error as well as script.
# 9  
Old 08-10-2010
Quote:
Originally Posted by pravin27
Please post error as well as script.
There is no error in the script you provided it is me who made the error when I wrote it. I had wrote num_type at the beginning and then sum_type for the print that is why I had no output.
Can you explain me what the script does exactly ?

Thanks.
# 10  
Old 08-10-2010
Hi,

Code:
#!/usr/bin/perl

$filename=shift;   ## First command line argument (log file name) will come in $filename
$sum_type=shift;   ## second command line argument (Summary Type)  will come in $sum_type
chomp($sum_type,$filename);  ## The chomp() function will remove (usually) any newline character from the end of a string. 			     
open(FH,"<","$filename") || die "cannot open file \n";  ## open file for read else exit from the script

while (<FH>) {
chomp;
if (/\s$sum_type\sSUMMARY\s\[(\w+)\.\w+\]\s(\w+)\scnt\/(\d+)\s/) { 
## pattern matching with log file record


#\w  Match "word" character (alphanumeric plus "_")
#\W  Match non-word character
#\s  Match whitespace character
#\S  Match non-whitespace character
#\d  Match digit character
#\D  Match non-digit character

#*      Match 0 or more times
#+      Match 1 or more times
#?      Match 1 or 0 times

print "$sum_type SUMMARY $1 $2 $3 \n";

#$1 -  If record match with pattern then whatever in first "()" (i.e. (\w+) ) is $1
#$2 -  If record match with pattern then whatever in second "()" (i.e. (\w+) ) is $2
#$3 -  If record match with pattern then whatever in third "()" (i.e. (\d+) ) is $3

 
}
}
close(FH);

This User Gave Thanks to pravin27 For This Post:
# 11  
Old 08-10-2010
Quote:
Originally Posted by pravin27
Hi,

Code:
#!/usr/bin/perl

$filename=shift;   ## First command line argument (log file name) will come in $filename
$sum_type=shift;   ## second command line argument (Summary Type)  will come in $sum_type
chomp($sum_type,$filename);  ## The chomp() function will remove (usually) any newline character from the end of a string. 			     
open(FH,"<","$filename") || die "cannot open file \n";  ## open file for read else exit from the script

while (<FH>) {
chomp;
if (/\s$sum_type\sSUMMARY\s\[(\w+)\.\w+\]\s(\w+)\scnt\/(\d+)\s/) { 
## pattern matching with log file record


#\w  Match "word" character (alphanumeric plus "_")
#\W  Match non-word character
#\s  Match whitespace character
#\S  Match non-whitespace character
#\d  Match digit character
#\D  Match non-digit character

#*      Match 0 or more times
#+      Match 1 or more times
#?      Match 1 or 0 times

print "$sum_type SUMMARY $1 $2 $3 \n";

#$1 -  If record match with pattern then whatever in first "()" (i.e. (\w+) ) is $1
#$2 -  If record match with pattern then whatever in second "()" (i.e. (\w+) ) is $2
#$3 -  If record match with pattern then whatever in third "()" (i.e. (\d+) ) is $3

 
}
}
close(FH);

A big thanks to you pravin. It is really clear now Smilie
# 12  
Old 08-10-2010
Hi Samb95

I can try to explain what pravin has done here.

Command line arguments are stored in the aray @ARGV in perl.

By doing a shift he is removing the elements and putting them in respective variables.

$firstname = shift;
$sum_type = shift;

Might i add its good practice to use "use strict" and "use warnings" modules and declare the variables preceded by a "my" keyword.

He then opens a file in read mode "<" and assigns it a file handler FH.

He loops through the file using FH. chomp removes the new line at the end of the input line read from the file.

Next you encounter the reg ex. looks formidable but simple. ( You should have a look at the reg exp tutorials ).

Since no argument is mentioned the reg ex acts on the $_ which contains the currernt input line of the file.

What he is doing is
1: \s$sum_type\sSUMMARY\s : looking for a space followed by your keyword followed by a space, the word SUMMARY followed by another space.

2: \[(\w+)\.\w+\]\s(\w+)\scnt\/(\d+)\s : escape the brackets to match an open square bracket.
Then look for one or more word characters (includes alphanumeric ) = \w+
Notice he has enclosed them in ( ) meaing it will be assigned to the perl special variable $1.

3: next he has matched a "." Since "." is a special character it has been escaped by a backslash. And the rest follow the similar analogy.

Hope this helps.

---------- Post updated at 10:27 AM ---------- Previous update was at 10:23 AM ----------

Just to be safe,

When ever you read command line arguments, your program should do checking to see if the user has entered the correct number of arguments. If not throw an error and quit the program.
This User Gave Thanks to abhijithtk For This Post:
# 13  
Old 08-10-2010
Quote:
Originally Posted by abhijithtk
Hi Samb95

I can try to explain what pravin has done here.

Command line arguments are stored in the aray @ARGV in perl.

By doing a shift he is removing the elements and putting them in respective variables.

$firstname = shift;
$sum_type = shift;

Might i add its good practice to use "use strict" and "use warnings" modules and declare the variables preceded by a "my" keyword.

He then opens a file in read mode "<" and assigns it a file handler FH.

He loops through the file using FH. chomp removes the new line at the end of the input line read from the file.

Next you encounter the reg ex. looks formidable but simple. ( You should have a look at the reg exp tutorials ).

Since no argument is mentioned the reg ex acts on the $_ which contains the currernt input line of the file.

What he is doing is
1: \s$sum_type\sSUMMARY\s : looking for a space followed by your keyword followed by a space, the word SUMMARY followed by another space.

2: \[(\w+)\.\w+\]\s(\w+)\scnt\/(\d+)\s : escape the brackets to match an open square bracket.
Then look for one or more word characters (includes alphanumeric ) = \w+
Notice he has enclosed them in ( ) meaing it will be assigned to the perl special variable $1.

3: next he has matched a "." Since "." is a special character it has been escaped by a backslash. And the rest follow the similar analogy.

Hope this helps.
Thanks for the advices and explanation abhijithtk, I will add them in the code.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parse log files

Hi all, We are having a sample log like .... test.log:2015.03.17 06:16:24 >> ABC.generateMethod() MethodAException while processing Request! DataForm: Header --- dtd: template.dtd, titleName: berger, requestId: 1503170032131, documentName: invoice123, hostName: acme.net, userName: userABC... (10 Replies)
Discussion started by: tandrei
10 Replies

2. Shell Programming and Scripting

awk Parse And Create Multiple Files Based on Field Value

Hello: I am working parsing a large input file which will be broken down into multiples based on the second field in the file, in this case: STORE. The idea is to create each file with the corresponding store number, for example: Report_$STORENUM_$DATETIMESTAMP , and obtaining the... (7 Replies)
Discussion started by: ec012
7 Replies

3. Shell Programming and Scripting

Perl script to parse all files in the folder

Hello Smart People! I have a perl script that will import xml data into an access db. I would like to modify it so it will automatcially parse through all xml files in the folder. I swa a post but couldnt get it working. her is what my scrip looks like, i only list the top if you need more... (3 Replies)
Discussion started by: cowboymaverick
3 Replies

4. Shell Programming and Scripting

Parse Log & Create Flow Diagram - Ideas/Tools

Hi, I would like to develop a script which parses the log file and generates a flow diagram ( graphical display ). We have an application, for understanding the sequence of functions call made, we have an debug line at "ENTRY/EXIT" of function. I have a small log parsing script which grep the... (2 Replies)
Discussion started by: ennstate
2 Replies

5. Shell Programming and Scripting

awk script to parse results from TWO files

I am trying to parse two files and get data that does not match in one of the columns ( column 3 in my case ) Data for two files are as follows A.txt ===== abc 10 5 0 1 16 xyz 16 1 1 0 18 efg 30 8 0 2 40 ijk 22 2 0 1 25 B.txt ===== abc... (6 Replies)
Discussion started by: roger67
6 Replies

6. UNIX for Dummies Questions & Answers

How can i parse my Unix log files??

Hello, i would like to parse Unix log files and i would like to use a Unix syslog analyzer. I'm going to use Eucalyptus and i would like to parse its log files. Is there any open source/free syslog parser?? Thanks, in advance! (2 Replies)
Discussion started by: g_p
2 Replies

7. Shell Programming and Scripting

Need a Script - Parse the log backup

Hello all, I need a script to parse the backup logs. I am newbie to scripting, please help. (7 Replies)
Discussion started by: sandeep007
7 Replies

8. Homework & Coursework Questions

shell script that can create, monitor the log files and report the issues for matching pattern

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: Write an automated shell program(s) that can create, monitor the log files and report the issues for matching... (0 Replies)
Discussion started by: itian2010
0 Replies

9. Shell Programming and Scripting

Help w/ script to read file and parse log message

Hi, I am working on the script to parsing the specific message like "aaaa" in multiple log files like N1-***,N2-***,N3-***... The script is to find the list of lof files which contains the message "aaaa" and export the list into excel filE. Can anyone give help? Thanks (2 Replies)
Discussion started by: shyork2001
2 Replies

10. Shell Programming and Scripting

shell scripts that parse log files

hi all ,i would like a shell script that parses log files and checks the contents for any anonalities,please help,thanks (4 Replies)
Discussion started by: trueman82
4 Replies
Login or Register to Ask a Question