Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Parsing a mixed format (flatfile+xml) logfile
Top Forums Shell Programming and Scripting Parsing a mixed format (flatfile+xml) logfile Post 302726529 by goddevil on Sunday 4th of November 2012 06:11:09 PM
Old 11-04-2012
Quote:
Originally Posted by Corona688
Code:
BEGIN {         OLDFS=FS="[()]" }
 
{
        for(X in XML) delete ARR[X];
        # Save some bits, and re-split line using <
        A=$1;   B=$3;   FS="<"; $0=$2
        for(N=1; N<=NF; N++)  # Process "tagname>data" strings only.
        {
                if($N == "")                    continue;
                if(substr($N,1,1) == "/")       continue; # Ignore close-tags
                if(split($N, ARR, ">") == 2)    XML[ARR[1]]=ARR[2];
        }
 
        # XML["event_n"] would be "blah" for example.
        for(X in XML) print X, XML[X];
 
        # Split on whitespace, dashes, and colons, and process the rest.
        FS="[ \r\n\t:-]+";      $0=A" "B
        # ...now available in $1 ... $N.
        print $1, $2, $3, $4, $5, $6, $7, $8
        FS=OLDFS        # So the next line splits on  ()
}

Code:
$ awk -f xml.awk datafile
 
$

Hi Corona,

I decided to give this a try as it is much more elagant than what i am doing.

On executing the script, i am getting the below error:
Code:
bash-3.00$ awk -f vtest.awk datafile > 1234
awk: XML is not an array
 record number 1

The contents of vtest.awk is:

Code:
BEGIN {         OLDFS=FS="[()]" }
{
        for(X in XML) delete ARR[X];
#        for(N=1; N<=4; N++) sub(/~/, "|");
        # Save some bits, and re-split line using <
        A=$1;   B=$3;   FS="<"; $0=$2
        for(N=1; N<=NF; N++)  # Process "tagname>data" strings only.
        {
                if($N == "")                    continue;
                if(substr($N,1,1) == "/")       continue; # Ignore close-tags
                if(split($N, ARR, ">") == 2)    XML[ARR[1]]=ARR[2];
        }
        for(X in XML) print X, XML[X];
        # Split on whitespace, dashes, and colons, and process the rest.
        FS="[ \r\n\t:-]+";      $0=A" "B
        # ...now available in $1 ... $N.
        print $1, $2, $3, $4, $5, $6, $7, $8
        FS=OLDFS        # So the next line splits on  ()
}
END{
print NR,"Records Processed";
}

Can you please assist me in what i am doing wrong?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

parsing xml

I want to use wget comment to parse an xml parse that exist in an online website. How can I connect it using shell script through Unix and how can I parse it?? (1 Reply)
Discussion started by: walnut
1 Replies

2. Shell Programming and Scripting

Logfile parsing with variable, multiple criterias among multiple lines

Hi all I've been working on a bash script parsing through debug/trace files and extracting all lines that relate to some search string. So far, it works pretty well. However, I am challenged by one requirement that is still open. What I want to do: 1) parse through a file and identify all... (3 Replies)
Discussion started by: reminder
3 Replies

3. Shell Programming and Scripting

logfile parsing

I thought I was pretty handy with awk until I got this one. :) I'm trying to parse a log file where the events could have different delimiters (2 scripts is ok), the errors are spread over multiple lines, and I"m trying to figure out how to not read the same lines that have already been read. ... (1 Reply)
Discussion started by: linkslice
1 Replies

4. Shell Programming and Scripting

XML parsing

I have a xml file attached. I need to parse parameterId and its value My output should be like 151515 38 151522 32769 and so on.. Please help me. Its urgent (6 Replies)
Discussion started by: LavanyaP
6 Replies

5. UNIX for Advanced & Expert Users

XML Parsing

I had a big XML and from which I have to make a layout as below *TOTAL+CB | *CB+FX | CS |*IR | *TOTAL | -------------------------------------------------------------------------------------------------- |CB FX | | | | DMFXNY EMSGFX... (6 Replies)
Discussion started by: manas_ranjan
6 Replies

6. Shell Programming and Scripting

Parsing XML

I am trying to parse an xml file and trying to grab certain values and inserting them into database table. I have the following xml that I am parsing: <dd:service name="locator" link="false"> <dd:activation mode="manual" /> <dd:run mode="direct_persistent" proxified="false" managed="true"... (7 Replies)
Discussion started by: $criptKid617
7 Replies

7. Shell Programming and Scripting

Generating XML from a flatfile

Hi all, I am trying to generate an XML file from a flatfile in ksh/bash (could also use perl at a pinch, but out of my depth there!). I have found several good solutions on this very forum for cases where the header line in the file forms the XML tags, however my flatfile is as follows:... (5 Replies)
Discussion started by: ianmrid
5 Replies

8. Shell Programming and Scripting

Parsing Logfile

Hi, I need to continuously monitor a logfile to get the log information between a process start and end. the logfile look like this abcdddddddddd cjjckkkkkkkkkkkk abc : Process started aaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbbb abc... (6 Replies)
Discussion started by: Byorg
6 Replies

9. Shell Programming and Scripting

XML: parsing of the Google contacts XML file

I am trying to parse the XML Google contact file using tools like xmllint and I even dived into the XSL Style Sheets using xsltproc but I get nowhere. I can not supply any sample file as it contains private data but you can download your own contacts using this script: #!/bin/sh # imports... (9 Replies)
Discussion started by: ripat
9 Replies

10. Shell Programming and Scripting

XML parsing

i have xml output in below format... <AlertsResponse> <Alert id="11216" name="fgdfg"> <AlertActionLog timestamp="1356521629778" user="admin" detail="Recovery Alert"/> </Alert> <Alert id="11215" name="gdfg <AlertActionLog timestamp="1356430119840" user=""... (12 Replies)
Discussion started by: vivek d r
12 Replies

LEARN ABOUT DEBIAN

lire::firewall::ipfilterdlfconverter

IpfilterDlfConverter(3pm)				  LogReport's Lire Documentation				 IpfilterDlfConverter(3pm)

NAME

       Lire::Firewall::IpfilterDlfConverter - convert ipf (ipmon) logs to firewall DLF

DESCRIPTION

       Lire::Firewall::IpfilterDlfConverter converts Ipfilter logs into firewall DLF format.  Input for this converter is the standard ipf syslog
       log file as produced by ipmon. IP Filter is shipped with FreeBSD, OpenBSD (up to 2.9) and some other OS's.

EXAMPLE

       A ipfilter logfile which looks like

	Oct 30 07:42:29 rolle ipmon[16747]: 07:42:28.585962	ie0 @0:9
	 b 192.168.48.1,45085 -> 192.168.48.2,22 PR tcp len 20 64 -S OUT
	Oct 30 07:40:24 rolle ipmon[16747]: 07:40:23.631307	ep1 @0:6
	 b 192.168.26.5,113 -> 192.168.26.1,3717 PR tcp len 20 40 -AR OUT
	Oct 30 07:42:29 rolle ipmon[16747]: 07:42:28.585962	ie0 @0:9
	 b 192.168.48.1,45085 -> 192.168.48.2,22 PR tcp len 20 64 -S OUT
	Oct 30 07:44:11 rolle ipmon[16747]: 07:44:10.605416 2x	   ep1 @0:15
	 b 192.168.26.1,138 -> 192.168.26.255,138 PR udp len 20 257  IN
	Oct 30 07:44:34 rolle ipmon[16747]: 07:44:33.891869	ie0 @0:10
	 b 192.168.48.1,23406 -> 192.168.48.2,22 PR tcp len 20 64 -S OUT
	Oct 30 07:49:13 rolle ipmon[16747]: 07:49:12.554420	ep1 @0:15
	 b 210.132.100.117 -> 192.168.26.5 PR icmp len 20 56 icmp 3/3 for
	 192.168.26.5,61915 - 210.132.100.117,53 PR udp len 20 23040 IN
	Oct 30 07:50:23 rolle ipmon[16747]: 07:50:22.908107	ep1 @0:15
	 b 210.132.100.117 -> 192.168.26.5 PR icmp len 20 56 icmp 3/3 for
	 192.168.26.5,4480 - 210.132.100.117,53 PR udp len 20 19712 IN
	Oct 30 07:56:11 rolle ipmon[16747]: 07:56:11.113029 2x	   ep1 @0:15
	 b 192.168.26.1,138 -> 192.168.26.255,138 PR udp len 20 257  IN

       (that's: .... 'PR' protocol 'len' length_of_ip_headers_saved packetlength direction) will get converted to something like

	994398737 denied igmp 100.187.115.1 - ep1 LIRE_NOTAVAIL 
	 224.0.0.2 - 56
	994398861 denied igmp 100.187.115.1 - ep1 LIRE_NOTAVAIL 
	 224.0.0.1 - 56
	994398862 denied igmp 100.187.115.1 - ep1 LIRE_NOTAVAIL 
	 224.0.0.2 - 56
	994406849 denied udp 192.168.26.4 137 ie0 LIRE_NOTAVAIL 
	 192.168.26.255 137 116
	994406850 denied udp 192.168.26.4 137 ie0 LIRE_NOTAVAIL 
	 192.168.26.255 137 116
	994406866 denied udp 192.168.26.4 137 ie0 LIRE_NOTAVAIL 
	 192.168.26.255 137 98

SEE ALSO

       ipl(4) for description of log structure.

       The ipmon.c source (e.g. on

	http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/ 
	 src/usr.sbin/ipmon/Attic/ipmon.c?rev=1.27& 
	 content-type=text/plain&hideattic=0

       ) for the specification of the log syntax.

       The IP Filter webpage on http://coombs.anu.edu.au/~avalon/ip-filter.html

AUTHOR

       Joost van Baal <joostvb@logreport.org>, Wessel Dankers <wsl@logreport.org>

VERSION

       $Id: IpfilterDlfConverter.pm,v 1.7 2009/03/15 08:10:55 vanbaal Exp $

COPYRIGHT

       Copyright (C) 2001-2003 Stichting LogReport Foundation LogReport@LogReport.org

       This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
       the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

       This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

       You should have received a copy of the GNU General Public License along with this program (see COPYING); if not, check with
       http://www.gnu.org/copyleft/gpl.html.

Lire 2.1.1							    2009-03-15						 IpfilterDlfConverter(3pm)
All times are GMT -4. The time now is 09:12 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy