Parse Logfile output variable


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parse Logfile output variable
# 1  
Old 12-09-2008
Parse Logfile output variable

Code:
<SUMMARY filecount_excluded="0" dirbytes_sent="3367893" dirbytes_hashcache="13275664" ..and so on..>
<session numthreads="1" type="avtarbackup" ndispatchers="1" ..and so on..><host numprocs="4" 
speed="900" osuser="root" name="ashsux01" memory="24545" /><build time="11:04:53" msgversion="13-10" 
appname="avtar" ..and so on../><dirstats numfiles="193129" numbytes="1461473417216" numdirs="0" />
</session><errorsummary exitcode="0" errors="0" warnings="0" fatals="0" />
</SUMMARY>



I would like to output the following:
Code:
SUMMARY_filecount="0"
SUMMARY_dirbytes="3367893"
...
SUMMARY_session_numthreads="1"
SUMMARY_session_type="avtarbackup" 
...
SUMMARY_session_build_time="11:04:53" 
SUMMARY_session_build_msgversion="13-10" 
...
SUMMARY_session_dirstats_numfiles="193129" 
SUMMARY_session_dirstats_numbytes="1461473417216" 
...
SUMMARY_errorsummary_exitcode="0" 
SUMMARY_errorsummary_errors="0" 
...

There could be any number of tags within tags and any number of variables.

I tried doing some searching but wasnt sure what exactally to search for.

Last edited by Ikon; 12-11-2008 at 05:36 PM..
# 2  
Old 12-09-2008
Tools Could you start with the following?

Code:
> cat file103 | sed "s/<\/session/~&/g" | tr "~" "\n"

Which would separate data onto each own line,
then grep for the lines you want to see?

[Sorry I couldn't test, but your sample data did not have the appropriate tags or session markers for me to truly analyze.]
# 3  
Old 12-09-2008
Here is an actual snip from the log:
Code:
<SUMMARY filecount_excluded="0" dirbytes_sent="3367893" dirbytes_hashcache="13275664"><session numthreads="1" 
type="avtarbackup" ndispatchers="1"><host numprocs="4" speed="900" osuser="root" name="ashsux01" memory="24545" />
<build time="11:04:53" msgversion="13-10" appname="avtar"/><dirstats numfiles="193129" numbytes="1461473417216" 
numdirs="0" /></session><errorsummary exitcode="0" errors="0" warnings="0" fatals="0" /></SUMMARY>

The problem im running into is <SUMMARY has its own vars then within SUMMARY it has SESSION and within SESSION it has its vars and then DIRSTATS, BUILD.

like this:
Code:
<SUMMARY VARS...>
	<SESSION VARS...>
		<BUILD VARS.../>
		<DIRSTATS VARS.../>
	</SESSION>
	<ERRORSUMMARY VARS...\>
</SUMMARY>

The actual log file has many others within summary and some not called summary they might be called something completely different than SUMMARY or SESSION.

Last edited by Ikon; 12-11-2008 at 05:52 PM..
# 4  
Old 12-09-2008
then i think you have to do it with two or more awk
i tried some thing but not getting right hope you work more on it
Code:
awk '/^<SUMMARY/{print $0}' file|awk -F"[=_]" 'BEGIN{RS=" "}$0!="<SUMMARY"{print "SUMMARY_"$1"="$3}'

Code:
awk '/^<session/{print $0}' file|awk -F"[= ]" 'BEGIN{RS=" "}$0!="<session"{print "SUMMARY_session_"$1"="$2}'

# 5  
Old 12-09-2008
Those are pretty goot starting points. They are pretty close.

I will be working on it all day tomorrow, when I get a working script i will be sure to share.
# 6  
Old 12-09-2008
Quote:
Originally Posted by Ikon
Those are pretty goot starting points. They are pretty close.

I will be working on it all day tomorrow, when I get a working script i will be sure to share.
will be looking forward for the answer..
regards,
vidya
# 7  
Old 12-11-2008
Well I decided to write it in perl...

Im having some problems:

If you compair the log with the output its skipping the first item, ie CUSTOMER_join, ADDRESS_street, TODAY_date.....

This is not the actual log file it will be reading it just a quick on I put together. The actual log if faily large.

Code:
 
# cat testfile.log
<CUSTOMER join="1/1/2008" last="12/20/2008">
<NAME name="John" lname="Smith">
<ADDRESS street="main" number="123" city="Orlando" state="Florida" zip="12345" />
</DATA>
</CUSTOMER>
<TODAY date="12/11/2008" time="12:12:12" />

Code:
# perl readlog.pl testfile.log
CUSTOMER_last: 12/20/2008
NAME_lname: Smith
ADDRESS_number: 123
ADDRESS_city: Orlando
ADDRESS_state: Florida
ADDRESS_zip: 12345
TODAY_time: 12:12:12

Code:
 
# cat readlog.pl
$filename = $ARGV[0];
open FILE, $filename or die $!;
while (<FILE>) {
        push(@fields, defined($1) ? $1:$3)
        while m/([^<>]+)/g;
}
close(FILE);
$head="";
foreach (@fields) {
        if ($_ =~ /^([A-Za-z0-9]+) /) {
                $line = $_;
                ($header,$data) = split(/ /, $line, 2);
                if ( $head == "" ) {
                        $head = $header;
                } else {
                        $head = $head."_".$header;
                }
                @subs = split(/" /,$data);
                for($i = 0; $i < @subs; $i++) {
                        ($str, $strdata) = split (/=/,$subs[$i]);
                        $strdata =~ s/^"//;
                        $strdata =~ s/"$//;
                        $head =~ s/_{2,}/_/;
                        if ($str !~ /\//) {
                                print $head."_".$str.": ".$strdata."\n";
                        }
                }
                if ($data =~ /\/$/) {
                        @h = split(/_/,$head);
                        $max = @h - 1;
                        $head =~ s/$h[$max]// ;
                }
        } else {
                if ($_ =~ /^\//) {
                         @h = split(/_/,$head);
                        $max = @h - 1;
                        $head =~ s/$h[$max]// ;
                }
        }
}


Last edited by Ikon; 12-11-2008 at 05:51 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Grepping for one variable while using awk to parse an associated variable

Im trying to search for a single variable in the first field and from that output use awk to extract out the lines that contain a value less than a value stored in another variable. Both the variables are associated with each other. Any guidance is appreciated. File that contains the... (6 Replies)
Discussion started by: ncwxpanther
6 Replies

2. Shell Programming and Scripting

ksh and Oracle stored procedure output in logfile

Friends, I pass some runtime arguments (date, number) through ksh script to Oracle procedure, use input value and pass it on to procedure. Oracle procedure gets input value, run query and logs everything in the logfile. I'm facing with couple of challenges 1. Even though I pass all... (5 Replies)
Discussion started by: homer4all
5 Replies

3. Shell Programming and Scripting

Logfile monitoring with logfile replacement

Bonjour, I've wrote a script to monitor a logfile in realtime. It is working almost perfeclty except for two things. The script use the following technique : tail -fn0 $logfile | \ while read line ; do ... some stuff done First one, I'd like a way to end the monitoring script if a... (3 Replies)
Discussion started by: Warluck
3 Replies

4. UNIX for Dummies Questions & Answers

Changes to write output to logfile and console

Friends, Below is the script which writes output to LOGFILE, however I want the entire log written to LOGFILE and also console. Please suggest me the changes I need to do here. #!/bin/ksh x=${0##*/} LOGFILE="x.log" echo "CAUTION : Files once deleted cannot be restored" printf 'Would... (8 Replies)
Discussion started by: fop4658
8 Replies

5. Shell Programming and Scripting

Echo cannot redirect first or second output to logfile

I created a script to do some work. I want to use "echo" to redirect "date" to log file. echo works to screen. But cannot redirect first or second "echo" output to logfile. Please help. My code looks like: STARTTIME=`date +%m-%d-%Y` LOGFILE=/directory/logfile.log echo "Start time:" $STARTTIME... (8 Replies)
Discussion started by: duke0001
8 Replies

6. Shell Programming and Scripting

Help about parse the variable

I'm using bash the code is QEMU_CMD="qemu-system-x86_64 -smp 2 -m 512 $QEMU_PARAMETER -hda $GUEST_IMAGE -kernel $GUEST_KERNEL -append \"root=/dev/hda rw console=ttyS0,115200 ip=$IP_PARAMETER \" -nographic" echo "..............................." echo "qemu command is... (9 Replies)
Discussion started by: yanglei_fage
9 Replies

7. Shell Programming and Scripting

Parse output path to set variable

I am looking to parse a text file output and set variables based on what is cropped from the parsing. Below is my script I am looking to add this feature too. All it does is scan a certain area of users directories for anyone using up more than X amount of disk space. It then writes to the... (4 Replies)
Discussion started by: es760
4 Replies

8. Shell Programming and Scripting

how to create a logfile to track the below script output

Hi Dudes, Can you please suggest me how to create a logfile to track the below script output ? Thanks #!/bin/ksh # backup the "std" I/P file descriptor exec 5<&0 #echo "Proceed ?" while read config_line; do # backup the I/P file descriptor of "while" block exec 6<&0 # restore the... (2 Replies)
Discussion started by: shirdi
2 Replies

9. Shell Programming and Scripting

how to parse value of the variable

I have a variable which has a full path to the file, for example : A=/t1/bin/f410pdb Does anybody know the command to parce this variable and assign the result to 3 other variables so each subdirectory name will be in a new variable like this B=t1 C=bin D=f410pdb Many thanks -A (5 Replies)
Discussion started by: aoussenko
5 Replies

10. Shell Programming and Scripting

parse variable

I have a variable (it is a date actually -> 2007-01-03) which would be passed in as parameter, what I want is to parse in and put year, month, and day in separate variables, I have tried the following but doesn't work echo $dt | awk -F- '{print $1 $2 $3}' | read y m d Thanks in... (2 Replies)
Discussion started by: mpang_
2 Replies
Login or Register to Ask a Question