Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Please Help with AWK to parse rapidly changing XML messages

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Please Help with AWK to parse rapidly changing XML messages
# 1  
Old 01-30-2012
Please Help with AWK to parse rapidly changing XML messages
Hi Guy's

Can I please get some help with this code.

I have xml feed file which rapidly changing temporary file and I need to capture the content of this file as soon as data arrives.

Example of the data

Quote:
[date+time], message=[DATA= "<?xml version="1.0?"><data changeMsg><NAME="John Smith"><Age="23"><D.O.B="11-10-1988"> <Gender="Male">"
[date+time], message=[DATA= "<?xml version="1.0?"><data changeMsg><NAME="Emy Williams"><Age="23"><D.O.B="01-05-1988"> <Gender="Female">"
[date+time], message=[DATA= "<?xml version="1.0?"><data changeMsg><NAME="Jack Adam"><Age="66"><D.O.B="24-07-1945"> <Gender="Male">"
[date+time], message=[DATA= "<?xml version="1.0?"><data changeMsg><NAME="Charlie Daniel"><Age="38"><D.O.B="15-08-1973"> <Gender="Male">"
[date+time], message=[DATA= "<?xml version="1.0?"><data changeMsg><NAME="Ruby James"><Age="38"><D.O.B="11-03-1973"> <Gender="Female">"
[date+time], message=[DATA= "<?xml version="1.0?"><data changeMsg><NAME="Sophie Thomas"><Age="20"><D.O.B="12-09-1991"><Gender="Female">"
Required data output
Quote:
8:30,Male,23,1
8:31,Female,23,1
8:32,Female,30,4
8:33,Male,50,10
Time is current time.

This is awk code that I have so far but this doesn't do what I need it to do. Can I please get help with it.

All I want the code to do is to run for 2 minutes process the counts , write it to output then do the same process again and again.

Code:
awk 'BEGIN { INTERVAL=120;    "date +%s"|getline sec;
    NEXT=sec+120;}

    {
        if(sec >= NEXT)
        {
           printf( "\nSummary\n" );
           for( x in agcount )
              printf( "%s,%d\n", x, agcount[x] ) | "sort";

           NEXT=sec+120;
        }

        gsub( ">", "" );        # strip uneeded junk and make "foo bar" easy to capture
        gsub( " ", "~" );
        gsub( "<", " " );

        for( i = 1; i <= NF; i++ )          # snarf up each name=value pair
        {
            if( split( $(i), a, "=" ) == 2 )
            {
                gsub(  "\"", "", a[2] );
                gsub(  "~", " ", a[2] );
                values[a[1]] = a[2];
            }
        }

        #gcount[values["Gender"]]++;         # collect counts
        #acount[values["Age"]]++;
        agcount[values["Gender"]","values["Age"]]++;

        printf( "%s %s %s %s\n", values["NAME"], values["Age"], values["D.O.B"], values["Gender"] );
    }' input-file

I can't use gawk or cron scheduler.

Will anyone be able to help me with this?

any help would be greatly appreciated.

James
# 2  
Old 01-30-2012
Don't you already have a thread about this?
# 3  
Old 01-31-2012
AWK will read the file once before executing. AWK doesn't treat the input file as Dynamic File as far as I know.

So why not put the awk in loop:

Code:
 
while [ 1 ]; do
 
awk 'BEGIN { INTERVAL=120;    "date +%s"|getline sec;
    NEXT=sec+120;}
 
    {
        if(sec >= NEXT)
        {
           printf( "\nSummary\n" );
           for( x in agcount )
              printf( "%s,%d\n", x, agcount[x] ) | "sort";
 
           NEXT=sec+120;
        }
 
        gsub( ">", "" );        # strip uneeded junk and make "foo bar" easy to capture
        gsub( " ", "~" );
        gsub( "<", " " );
 
        for( i = 1; i <= NF; i++ )          # snarf up each name=value pair
        {
            if( split( $(i), a, "=" ) == 2 )
            {
                gsub(  "\"", "", a[2] );
                gsub(  "~", " ", a[2] );
                values[a[1]] = a[2];
            }
        }
 
        #gcount[values["Gender"]]++;         # collect counts
        #acount[values["Age"]]++;
        agcount[values["Gender"]","values["Age"]]++;
 
        printf( "%s %s %s %s\n", values["NAME"], values["Age"], values["D.O.B"], values["Gender"] );
    }' input-file
 
   ## Keep one LOOP BREAK CONDITION if you need to break out of loop on certain exception or conditions here
 
  if [ BREAKLOOP ]; then
       break
  fi
 
done

# 4  
Old 01-31-2012
He already has a thread about this, wherein the rapidly-changing file has been replaced with a pipe which causes it to be read continually, which is an improvement on it not being read at all.
# 5  
Old 02-02-2012
The code I have is not doing what is should do the print is not printing to the output file and the 2 minute loop doesn't work at all.

@knight_eon

The section of the code is not working for some reason, is this something you can help with?
Code:
'BEGIN { INTERVAL=120;    "date +%s"|getline sec;
    NEXT=sec+120;}
 
    {
        if(sec >= NEXT)
        {
           printf( "\nSummary\n" );
           for( x in agcount )
              printf( "%s,%d\n", x, agcount[x] ) | "sort";
 
           NEXT=sec+120;
        }

Could anyone please help with this.

Thanks
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk Script to parse a XML tag

I have an XML tag like this: <property name="agent" value="/var/tmp/root/eclipse" /> Is there way using awk that i can get the value from the above tag. So the output should be: /var/tmp/root/eclipse Help will be appreciated. Regards, Adi (6 Replies)
Discussion started by: asirohi
6 Replies

2. Shell Programming and Scripting

AWK to Parse XML messages

Hello Guys, Please help with AWK problem. I have XML file which contains a list of messages for subjects. Example of the messages: , message=, message=, message=, message=, message=, message=, message=, message= I want to use AWK to parse these xml messages but I am... (7 Replies)
Discussion started by: James_Owen
7 Replies

3. Shell Programming and Scripting

python - wget xml doc and parse with awk

Well, that's what I'd do in bash :) Here's what I have so far: import urllib2 from BeautifulSoup import BeautifulStoneSoup xml = urllib2.urlopen('http://weatherlink.com/xml.php?user=blah&pass=blah') soup = BeautifulStoneSoup(xml) print soup.prettify() but all it does is grab the html... (0 Replies)
Discussion started by: unclecameron
0 Replies

4. Shell Programming and Scripting

Shell script (not Perl) to parse xml with awk

Hi, I have to make an script according to these: - I have couples of files like: xxxxxxxxxxxxx.csv xxxxxxxxxxxxx_desc.xml - every xml file has diferent fields, but keeps this format: ........ <defaultName>2011-02-25T16:43:43.582Z</defaultName> ........... (2 Replies)
Discussion started by: Pluff
2 Replies

5. Shell Programming and Scripting

Parse XML

Hi all! I'm looking to write a quick script and in it I need to request an XML file from a service running on localhost and parse that XML file and output it. I'm looking to do it in bash although it doesn't really matter what shell it is in. The XML file returned would look like this: ... (3 Replies)
Discussion started by: mtehonica
3 Replies

6. UNIX for Dummies Questions & Answers

Parse XML e report con AWK

Thanks in advance who can answer. I have to make a small shell that after reading an XML file and extract the fields I create a text file with the same fields taken previously in tabular form. I did this parse.awk ---------------------- BEGIN { FS="" } { for(i=2; i<=NF; i+=2) { ... (1 Reply)
Discussion started by: mcarlo65
1 Replies

7. Shell Programming and Scripting

how to parse the file in xml format using awk/nawk

Hi All, I have an xml file with the below format. <a>111</a><b>222</b><c>333<c><d><e>123</e><f>234</f><d><e>456</e><f>789</f> output needed is 111,222,333,123,234 111,222,333,456,789 nawk 'BEGIN{FS="<|>"} {print a,b,c,e,f a="" ... (7 Replies)
Discussion started by: natalie23
7 Replies

8. Shell Programming and Scripting

Parse an XML task list to create each task.xml file

I have an task definition listing xml file that contains a list of tasks such as <TASKLIST <TASK definition="Completion date" id="Taskname1" Some other <CODE name="Code12" <Parameter pname="Dog" input="5.6" units="feet" etc /Parameter> <Parameter... (3 Replies)
Discussion started by: MissI
3 Replies

9. Shell Programming and Scripting

Need AWk To parse XML logs

Hi , I need an Awk script to parse my log file . 2008-04-26 10:00:13,391 INFO Logger - <?xml version="1.0" encoding="UTF-8" standalone="no"?><2dm tmsg... (0 Replies)
Discussion started by: amit1_x
0 Replies

10. Shell Programming and Scripting

How to parse a XML file using PERL and XML::DOm

I need to know the way. I have got parsing down some nodes. But I was unable to get the child node perfectly. If you have code please send it. It will be very useful for me. (0 Replies)
Discussion started by: girigopal
0 Replies
Login or Register to Ask a Question