Logfile parsing with variable, multiple criterias among multiple lines
Hi all
I've been working on a bash script parsing through debug/trace files and extracting all lines that relate to some search string. So far, it works pretty well. However, I am challenged by one requirement that is still open.
What I want to do:
1) parse through a file and identify all packet numbers (PXXX:) that match my search, hereafter called "interesting packets"
2) parse again through the same file, searching this time now for packets that relate to the packets identified in step 1)
See note around P3712451 in the example below!
3) what I would also like to get are related log messages that may appear just underneath a interesting packet. Any other log message should be ignored.
4) output all log file lines that somehow relate to the searched string into another file.
Example trace file (simplified):
Code:
12/14/2009 21:16:03: P3712446: Packet received from 10.10.10.1
12/14/2009 21:16:03: P3712446: Trace of Accounting-Request packet
12/14/2009 21:16:03: P3712446: identifier = 33
12/14/2009 21:16:03: P3712446: length = 435
12/14/2009 21:16:03: P3712446: NAS-Port = 1
12/14/2009 21:16:03: P3712446: Service-Type = Framed
12/14/2009 21:16:03: P3712446: Framed-Protocol = PPP
12/14/2009 21:16:03: P3712446: NAS-Port-Type = Virtual
12/14/2009 21:16:03: P3712446: User-Name = testuser
12/14/2009 21:16:03: P3712446-2: Creating proxy request P3712451 to send to RemoteServer rsAAA1 (11.11.11.11) <==== P3712451 is related to P3712446
12/14/2009 21:16:03: P3712451: Trace of Accounting-Request packet
12/14/2009 21:16:03: P3712451: identifier = 33
12/14/2009 21:16:03: P3712451: length = 435
12/14/2009 21:16:03: P3712451: NAS-Port = 1
12/14/2009 21:16:03: P3712451: Service-Type = Framed
12/14/2009 21:16:03: P3712451: Framed-Protocol = PPP
12/14/2009 21:16:03: P3712451: NAS-Port-Type = Virtual
12/14/2009 21:16:03: P3712451: User-Name = testuser
12/14/2009 21:16:04: P3712460: Packet received from 11.11.11.11
12/14/2009 21:16:04: Log: Positive response received from 11.11.11.11 <===== log message that should be captured as well
12/14/2009 21:16:04: P3712446-2: Creating response from proxy response P3712460
12/14/2009 21:16:04: P3712446-2: Sub-service REMOTEAAA accepted request
12/14/2009 21:16:04: P3712446: All sub-services accepted the request
12/14/2009 21:16:04: P3712446: Trace of Accounting-Response packet
12/14/2009 21:16:04: P3712446: identifier = 33
12/14/2009 21:16:04: P3712446: length = 20
12/14/2009 21:16:04: P3712446: Sending response to 10.10.10.1
Step 1), 2) and 4) are already working using egrep.
Step 1)
I've got 2 questions now:
Q1) How can I catch Log lines like...
Code:
12/14/2009 21:16:04: Log: Positive response received from 11.11.11.11
...if it follows an interesting packet and ignore any other Log line?
I've been looking at multiple line matching examples... but I am not able to apply what I've seen in combination with the sometimes huge list of interesting packets I've got.
Q2) Any obvious and easy way to simplify what I've done already?
I started with parsing each line... but that was far too time consuming (1h+). The above still takes 2-3 minutes for a 130MB file, which is ok. But maybe someone has even something faster on his mind.
I've been told that sed is a "Turing complete" language, whatever the bleep that means... so programmatically speaking, you could do the whole thing in sed.
I prefer just using ksh. Something like this:
Code:
print_me=1
egrep "($QUERYSTRING|Log:)" logfile |
while read junk ; do
if [[ $print_me -eq 1 ]]; then
print "$junk"
fi
if [[ $junk = "*Log*" ]]; then
print_me=0
else
print_me=1
fi
done
Many thanks for this.
Problem is that all log messages would follow an interesting packet with the initial egrep in your example. Hence too many irrelevant log messages would be displayed.
Or do I miss something?
Any other idea how the 4 steps and step 3 in particular could be realized?
Seems I need to refine my problem. I've probably provided too much background information.
Assuming a file as follows
Code:
this is line with interesting content1
this is another line with interesting content1
this is a log line that is related to the interesting message above and should be shown!
this is line with uninteresting content2
this is another line with uninteresting content2
this is a log line that is related to the UNinteresting message above and should NOT be shown!
this is line with interesting content3
this is another line with interesting content3
this is a log line that is related to the interesting message above and should be shown!
Assuming I've got a list of interesting content, in my example above "(content1|content3)", how can I extract all lines with interesting content as well as their related log lines just underneath?
Hello Gurus,
I have a multiple pipe separated files which have records going over multiple Lines. End of line separator is \n and records going over multiple lines have <CR> as separator. below is example from one file.
1|ABC DEF|100|10
2|PQ
RS
T|200|20
3| UVWXYZ|300|30
4| GHIJKL|400|40... (7 Replies)
from the CLI on a Mac, if you type networksetup -listallnetworkservices then you get results in a multi-line paragraph that look something like this:
networksetup -listallnetworkservices
An asterisk (*) denotes that a network service is disabled.
Wi-Fi
Display Ethernet
Bluetooth DUN... (7 Replies)
GM,
I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed.
I am assuming that sed, awk or even perl could do what I need.
I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Hello,
I would like to ask for help with csh script.
An example of an input in .txt file is below, the number of lines varies from file to file and I have 2 or 3 columns with values. I would like to read all the values (probably one by one) and set them to independent unique variables that... (7 Replies)
Good Morning/Afternoon All,
I am having some trouble creating a variable called "total" to display the sum of the values in a specific field, $6 for example.
The data I am working on is in the following form:
John Doe:(555) 555-5555:1:2:3
Jane Doe:(544) 444-5556:4:5:6
Moe Doe:(654)... (2 Replies)
I have a file that contains the following lines
the brown quick fox
jumped over
the white laze dog
0123456789
I wanted to put the contents of this file into a variable so I used this code:
VAR_LIST=`cat $2`
where $2 is the file name passed as an argument to the script
If I... (3 Replies)
I have a file with a set of insert statements some of which have a single column value that crosses multiple lines causing the statement to fail in sql*plue. Can someone help me with a sed script to replace the new lines with chr(10)?
here is an example:
insert into mytable(id, field1, field2)... (3 Replies)
Good morning,
Novice scripter in Unix here, and I've run into and sed task I can't quite wrap my head around. I'm pulling my hair out fast enough as it is and thought I would go to the knowledge bank.
I have a sorted file that I'm trying to trim down by deleting any line whose first few... (2 Replies)
