Parse out known messages from a log file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parse out known messages from a log file
# 1  
Old 10-01-2008
Parse out known messages from a log file

I am looking for a script to do the following. I have a large log file that contains hundreds of warnings, a lot of which can be ignored. The tool doesn't allow me to suppress it, so I like to parse it out from the log file and isolate just the new messages/warnings, based on an exception file.

I have three files,
- log1.txt - first log file generated by the tool.
- log1.exp which is an exception list file that I will create by cutting and pasting items from log1.txt, that can be exempted/ignored from future log files.
- log2.txt will be the new log file containing all or some of the items in the log1.txt and possibly a bunch of new warnings, the ones that I am after.

%>cat log1.txt (first log file)
Date: Sep 11, 2008 10:30 PM
Tool: Micorsoft SQL Server

The signal "/RAM12/dout<15>" is sourceless and has been removed.
The signal "/RAM22/dout<14>" is sourceless and has been removed.
The signal "/RAM23/dout<13>" is sourceless and has been removed.
The signal "/RAM24/dout<12>" is sourceless and has been removed.
Sourceless block
"U0/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U1/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U2/blk_generator/valid/has_B/dout1" (SFF) removed.
Sourceless block
"U3/blk_generator/valid/has_B/dout1" (SFF) removed.
...
..
and a whole bunch of other messages.


%>cat log2.txt will also display similar text with few changes like the date and any messages that were never found after the first run.


%>cat log1.exp (Exception list manually created using text from log1.txt)
The signal "/RAM12/dout<15>" is sourceless and has been removed.
The signal "/RAM22/dout<14>" is sourceless and has been removed.
The signal "/RAM23/dout<13>" is sourceless and has been removed.
The signal "/RAM24/dout<12>" is sourceless and has been removed.


>cleanScript.pl log2.txt log1.exp
Date: Sep 13, 2008 11:45AM
Tool: Micorsoft SQL Server

Sourceless block
"U0/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U1/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U2/blk_generator/valid/has_B/dout1" (SFF) removed.
Sourceless block
"U3/blk_generator/valid/has_B/dout1" (SFF) removed.
...
..
and a whole bunch of other messages.


Any help would be greatly appreciated. Perl or AWk would be great.

Basically the cleanScript.pl script would parse through log2.txt, delete any entries that are listed in log1.exp and write out a new file. Please note that some messages are wrapped between multiple lines, like the "Sourceless blok.." warning.

Last edited by cdn2008; 10-01-2008 at 01:56 PM..
# 2  
Old 10-01-2008
Can anyone please help. If you need more clarification, please let me know. I hope I am not in the wrong forum for this type of questions.
# 3  
Old 10-01-2008
maybe I'm over simplifying here...but couldn't you just do something like

Code:
egrep -vf what.you.want.excluded file.with.whole.log > newfile.with.trimmed.list

# 4  
Old 10-01-2008
Unfortunately that won't do it. That, as you said is over simplified. I tried it, but gets an empty file.
# 5  
Old 10-01-2008
Code:
d@DeCobox-Micro ~
$ cat > orig
The signal "/RAM12/dout<15>" is sourceless and has been removed.
The signal "/RAM22/dout<14>" is sourceless and has been removed.
The signal "/RAM23/dout<13>" is sourceless and has been removed.
The signal "/RAM24/dout<12>" is sourceless and has been removed.
Sourceless block
"U0/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U1/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U2/blk_generator/valid/has_B/dout1" (SFF) removed.
Sourceless block
"U3/blk_generator/valid/has_B/dout1" (SFF) removed

d@DeCobox-Micro ~
$ cat > excep
The signal "/RAM12/dout<15>" is sourceless and has been removed.
The signal "/RAM22/dout<14>" is sourceless and has been removed.
The signal "/RAM23/dout<13>" is sourceless and has been removed.
The signal "/RAM24/dout<12>" is sourceless and has been removed.

d@DeCobox-Micro ~
$ egrep -vf excep orig > happydays

d@DeCobox-Micro ~
$ cat happydays 
Sourceless block
"U0/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U1/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U2/blk_generator/valid/has_B/dout1" (SFF) removed.
Sourceless block
"U3/blk_generator/valid/has_B/dout1" (SFF) removed

d@DeCobox-Micro ~
$

# 6  
Old 10-02-2008
That is strange...
I get an empty happydays file. I am running on Linux RHE4.0. Would that make a difference.
Code:
zh0tx-.../temp < 114> cat excep 
The signal "/RAM12/dout<15>" is sourceless and has been removed.
The signal "/RAM22/dout<14>" is sourceless and has been removed.
The signal "/RAM23/dout<13>" is sourceless and has been removed.
The signal "/RAM24/dout<12>" is sourceless and has been removed.

zh0tx-.../temp < 115> cat orig 
The signal "/RAM12/dout<15>" is sourceless and has been removed.
The signal "/RAM22/dout<14>" is sourceless and has been removed.
The signal "/RAM23/dout<13>" is sourceless and has been removed.
The signal "/RAM24/dout<12>" is sourceless and has been removed.
Sourceless block
"U0/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U1/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U2/blk_generator/valid/has_B/dout1" (SFF) removed.
Sourceless block
"U3/blk_generator/valid/has_B/dout1" (SFF) removed

zh0tx-.../temp < 116> egrep -vf excep orig > happydays
zh0tx-.../temp < 117> cat happydays
zh0tx-.../temp < 118>

# 7  
Old 10-02-2008
Quote:
Originally Posted by cdn2008
That is strange...
I get an empty happydays file. I am running on Linux RHE4.0. Would that make a difference.
Code:
zh0tx-.../temp < 114> cat excep 
The signal "/RAM12/dout<15>" is sourceless and has been removed.
The signal "/RAM22/dout<14>" is sourceless and has been removed.
The signal "/RAM23/dout<13>" is sourceless and has been removed.
The signal "/RAM24/dout<12>" is sourceless and has been removed.

zh0tx-.../temp < 115> cat orig 
The signal "/RAM12/dout<15>" is sourceless and has been removed.
The signal "/RAM22/dout<14>" is sourceless and has been removed.
The signal "/RAM23/dout<13>" is sourceless and has been removed.
The signal "/RAM24/dout<12>" is sourceless and has been removed.
Sourceless block
"U0/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U1/blk_generator/valid/has_B/dout0" (SFF) removed.
Sourceless block
"U2/blk_generator/valid/has_B/dout1" (SFF) removed.
Sourceless block
"U3/blk_generator/valid/has_B/dout1" (SFF) removed

zh0tx-.../temp < 116> egrep -vf excep orig > happydays
zh0tx-.../temp < 117> cat happydays
zh0tx-.../temp < 118>

Well now I'm interested to find out what the deal is. My first post was in Cygwin...but to be sure I just VPN'd to a Solaris system & a Linux system at work and got the same results. I look forward to finding out what's up.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Parse apache log file with three different time formats

Hi, I want to parse below file and Write a function to extract the logs between two given timestamp. Apache (Unix) Log Samples - MonitorWare The challenge here is there are three date and time format. First :- 07/Mar/2004:16:05:49 Second :- Sun Mar 7 16:02:00 2004 Third :- 29-Mar... (6 Replies)
Discussion started by: sahil_shine
6 Replies

2. Shell Programming and Scripting

Shell Script | Parse log file after a given date and time stamp

I am developing one script which will take log file name, output file name, date, hour and minute as an argument and based on these inputs, the script will scan and capture all the error(s) that have been triggered from a given time. Example: script should capture all the error after 13:50 on Jan... (2 Replies)
Discussion started by: ROMA3
2 Replies

3. Shell Programming and Scripting

Parse log file to insert into database

I have a log file that's created daily by this command: sar -u 300 288 >> /var/log/usage/$(date "+%Y-%m-%d")_$(hostname)_cpu.log It that contains data like this: Linux 3.16.0-4-amd64 (myhostname) 08/15/2015 _x86_64_ (1 CPU) 11:34:17 PM CPU %user %nice ... (12 Replies)
Discussion started by: unplugme71
12 Replies

4. Shell Programming and Scripting

Parse A Log File

Hello All, Below is the excerpt from my Informatica log file which has 4 blocks of lines (starting with WRITER_1_*_1). Like these my log file will have multiple blocks of same pattern. WRITER_1_*_1> WRT_8161 TARGET BASED COMMIT POINT Thu May 08 09:33:21 2014... (13 Replies)
Discussion started by: Ariean
13 Replies

5. Shell Programming and Scripting

Check/Parse log file's lines using time difference/timestamp

I was looking at this script which outputs the two lines which differs less than one sec. #!/usr/bin/perl -w use strict; use warnings; use Time::Local; use constant SEC_MILIC => 1000; my $file='infile'; ## Open for reading argument file. open my $fh, "<", $file or die "Cannot... (1 Reply)
Discussion started by: cele_82
1 Replies

6. Shell Programming and Scripting

Sed to parse log file

Hi all, thanks for reading the post. I'm trying to parse hundreds of log files in a directory. One log file looks similar to below: Investigator : Jim_Foo Custodian : Jim_Foo-HDD1-FOO-1234 Export Path : N:\FOO-1234\Foo_Foo Compute MD5 : No File List Only: No Extensions Selected:... (4 Replies)
Discussion started by: chipperuga
4 Replies

7. Shell Programming and Scripting

Parse the log file

./abc.sh started at Sun Oct 24 06:42:04 PDT 2010 Message: ======= Summary Report of NAME count ----------------------------------------------------------------- Below is the output of the SQL query :- NAME COUNT... (2 Replies)
Discussion started by: sandy1028
2 Replies

8. Shell Programming and Scripting

parse a log file and remember last line

Hi all: I'm working on a HPUX 11.23 system and I am needing to parse a tomcat-jakarta log file for memory use. Getting the desired data is easy, assuming the log file does not grow. This file grows constantly and I want to check it q 5 min. The next check will pick up from where it left off 5... (4 Replies)
Discussion started by: raggmopp
4 Replies

9. Shell Programming and Scripting

Help w/ script to read file and parse log message

Hi, I am working on the script to parsing the specific message like "aaaa" in multiple log files like N1-***,N2-***,N3-***... The script is to find the list of lof files which contains the message "aaaa" and export the list into excel filE. Can anyone give help? Thanks (2 Replies)
Discussion started by: shyork2001
2 Replies

10. Shell Programming and Scripting

Piping tail to awk to parse a log file

Hello all, I've got what I'm pretty sure is a simple problem, but I just can't seem to work past it. I'm trying to use awk to pretty up a log file, and calculate a percentage. The log file looks like this: # tail strtovrUsage 20090531-18:15:45 RSreq - 24, RSsuc - 24, RSrun - 78, RSerr -... (4 Replies)
Discussion started by: DeCoTwc
4 Replies
Login or Register to Ask a Question