Sponsored Content
Full Discussion: Parsing syslog from Linux
Top Forums Shell Programming and Scripting Parsing syslog from Linux Post 303037028 by RudiC on Sunday 21st of July 2019 03:33:03 AM
Old 07-21-2019
The code has to match NF fields against 9 items for every line; this will take its time, esp. on large files. I compared (timed) your code to mine on a medium sized sample data file and found that yours is roughly two to three times slower, so I don't understand the 27 min of my code vs. 6 min of your code. Still, going through my proposal again and trying to tease out a few percent, I came up with
Code:
awk '
BEGIN   {print HDLN = "eventtime|srcip|dstip|srcport|dstport|transip|transport|action|sessionid"
         MX = split (HDLN, HD, "|")
         for (i=1; i<=MX; i++) L[i] = length (HD[i]) + 1
        }
        {OUT = DL = ""
         for (i=1; i<=MX; i++)  {match ($0, HD[i] "=[^ ]*")
                                 OUT = OUT DL  substr ($0, RSTART + L[i], RLENGTH - L[i])
                                 DL = "|"
                                }
         print OUT 
        }
' file

Pls try and report back, esp. in comparison to your code in post #5 (don't forget you'll need to match the fields' sequence to the header's).
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need some help with parsing

I have a big xml file with little formatting in it. It contains over 600 messages that I need to break each message out in its own separate file. The xml file looks in the middle of it something like this: </Title></Msg><Msg><Opener> Hello how are you?<Title> Some says hello</Title><Body>... (3 Replies)
Discussion started by: quixoticking11
3 Replies

2. Shell Programming and Scripting

Perl parsing compared to Ksh parsing

#! /usr/local/bin/perl -w $ip = "$ARGV"; $rw = "$ARGV"; $snmpg = "/usr/local/bin/snmpbulkget -v2c -Cn1 -Cn2 -Os -c $rw"; $snmpw = "/usr/local/bin/snmpwalk -Os -c $rw"; $syst=`$snmpg $ip system sysName sysObjectID`; sysDescr.0 = STRING: Cisco Internetwork Operating System Software... (1 Reply)
Discussion started by: popeye
1 Replies

3. Shell Programming and Scripting

Parsing of file for Report Generation (String parsing and splitting)

Hey guys, I have this file generated by me... i want to create some HTML output from it. The problem is that i am really confused about how do I go about reading the file. The file is in the following format: TID1 Name1 ATime=xx AResult=yyy AExpected=yyy BTime=xx BResult=yyy... (8 Replies)
Discussion started by: umar.shaikh
8 Replies

4. Red Hat

Parsing a linux file and formatting it.

Hi, I have a linux file that has data like this.. REQUEST_ID|text^Ctext^Ctext^C REQUEST_ID|text^Ctext^C REQUEST_ID| REQUEST_ID| REQUEST_ID|text^Ctext^Ctext^Ctext^Ctext^Ctext^C.... Where ever I see a ^C character, I need to copy the corresponding REQUEST_ID and that part of the text to a new... (17 Replies)
Discussion started by: charithainfadev
17 Replies

5. Shell Programming and Scripting

Parsing kiwi syslog from Astaro

Hello, I am trying to parse this syslog pulling out and logging results to a file. The information I want is: scrip, scrport, dstip, dstport. I just want the numbers, not including the text part ie srcip=". Problem is, the column locations change, so I can't use the nice awk $1 $2 etc to... (4 Replies)
Discussion started by: rmelnik
4 Replies

6. UNIX for Dummies Questions & Answers

Parsing linux commands through FTP

Hi Techies, I have made a shell script which stores the output of it in a text file. then i wanted to fetch that text file using windows scheduler in my windows xp desktop which i did successfully using the below mentioned ftp .bat file : @echo off @echo ftp_user>ftp_test.scr @echo... (0 Replies)
Discussion started by: gemnian.g
0 Replies

7. Shell Programming and Scripting

Help - Parsing data in XML in Linux

Hi, I have an XML file in Linux and it contains a long string of characters. The last part of the file is like ....... ....... ....... CAD</MarketDescription></InvestorTransaction></AdvisorAccount></DivisionAdvisor></Division>... (3 Replies)
Discussion started by: naveed
3 Replies

8. Shell Programming and Scripting

Specific string parsing in Linux/UNIX

Hi, I have a string which can be completely unstructred. I am looking to parse out values within that String. Here is an example <Random Strings> String1=<some number a> String2=<some number b> String3=<some number c> Satish=<some number d> String4=<some number e> I only want to parse out... (1 Reply)
Discussion started by: satishrao
1 Replies

9. SuSE

Location and name of SYSLOG in SUSE Linux

Esteemed listers, Where is the location of SYSLOG file? In etc/auditd.conf script, the log_file location is '/var/log/audit/audit.log' as below. Is this the location where SYSLOG is stored? Thank you in advance, log_file = /var/log/audit/audit.log log_format = RAW... (3 Replies)
Discussion started by: JDBA
3 Replies

10. Programming

Openlog and syslog in red-hat Linux doesn't write any thing to /var/log/*

Using redhat 64 bit ver 6.2 I have simple c++ app that is trying to write to syslog like this: /* try to write massage into linux log */ void foo::writeToSyslog() { openlog("testlogfoo", 0, 24); // Send the message. ... (1 Reply)
Discussion started by: umen
1 Replies
Time::Progress(3)					User Contributed Perl Documentation					 Time::Progress(3)

NAME
Time::Progress - Elapsed and estimated finish time reporting. SYNOPSIS
use Time::Progress; # autoflush to get working $| = 1; # get new `timer' my $p = new Time::Progress; # restart and report progress $p->restart; sleep 5; # or do some work here print $p->report( "done %p elapsed: %L (%l sec), ETA %E (%e sec) ", 50 ); # set min and max values $p->attr( min => 2, max => 20 ); # restart `timer' $p->restart; my $c; for( $c = 2; $c <= 20; $c++ ) { # print progress bar and percentage done print $p->report( "eta: %E min, %40b %p ", $c ); sleep 1; # work... } # stop timer $p->stop; # report times print $p->elapsed_str; DESCRIPTION
Shortest time interval that can be measured is 1 second. The available methods are: new my $p = new Time::Progress; Returns new object of Time::Progress class and starts the timer. It also sets min and max values to 0 and 100, so the next report calls will default to percents range. restart restarts the timer and clears the stop mark. optionally restart() may act also as attr() for setting attributes: $p->restart( min => 1, max => 5 ); is the same as: $p->attr( min => 1, max => 5 ); $p->restart(); If you need to count things, you can set just 'max' attribute since 'min' is already set to 0 when object is constructed by new(): $p->restart( max => 42 ); stop Sets the stop mark. This is only useful if you do some work, then finish, then do some work that shouldn't be timed and finally report. Something like: $p->restart; # do some work here... $p->stop; # do some post-work here print $p->report; # `post-work' will not be timed Stop is useless if you want to report time as soon as work is finished like: $p->restart; # do some work here... print $p->report; continue Clears the stop mark. (mostly useless, perhaps you need to restart?) attr Sets and returns internal values for attributes. Available attributes are: min This is the min value of the items that will follow (used to calculate estimated finish time) max This is the max value of all items in the even (also used to calculate estimated finish time) format This is the default report format. It is used if report is called without parameters. attr returns array of the set attributes: my ( $new_min, $new_max ) = $p->attr( min => 1, max => 5 ); If you want just to get values use undef: my $old_format = $p->attr( format => undef ); This way of handling attributes is a bit heavy but saves a lot of attribute handling functions. attr will complain if you pass odd number of parameters. report report is the most complex method in this package. :) expected arguments are: $p->report( format, [current_item] ); format is string that will be used for the result string. Recognized special sequences are: %l elapsed seconds %L elapsed time in minutes in format MM:SS %e remaining seconds %E remaining time in minutes in format MM:SS %p percentage done in format PPP.P% %f estimated finish time in format returned by localtime() %b %B progress bar which looks like: ##############...................... %b takes optional width: %40b -- 40-chars wide bar %9b -- 9-chars wide bar %b -- 79-chars wide bar (default) Parameters can be ommited and then default format set with attr will be used. Sequences 'L', 'l', 'E' and 'e' can have width also: %10e %5l ... Estimate time calculations can be used only if min and max values are set (see attr method) and current item is passed to report! if you want to use the default format but still have estimates use it like this: $p->format( undef, 45 ); If you don't give current item (step) or didn't set proper min/max value then all estimate sequences will have value `n/a'. You can freely mix reports during the same event. elapsed estimate helpers -- return elapsed/estimate seconds. elapsed_str estimate_str helpers -- return elapsed/estimated string in format: "elapsed time is MM:SS min. " "remaining time is MM:SS min. " all helpers need one argument -- current item. FORMAT EXAMPLES
# $c is current element (step) reached # for the examples: min = 0, max = 100, $c = 33.3 print $p->report( "done %p elapsed: %L (%l sec), ETA %E (%e sec) ", $c ); # prints: # done 33.3% elapsed time 0:05 (5 sec), ETA 0:07 (7 sec) print $p->report( "%45b %p ", $c ); # prints: # ###############.............................. 33.3% print $p->report( "done %p ETA %f ", $c ); # prints: # done 33.3% ETA Sun Oct 21 16:50:57 2001 GITHUB REPOSITORY
git@github.com:cade-vs/perl-time-progress.git git clone git://github.com/cade-vs/perl-time-progress.git AUTHOR
Vladi Belperchinov-Shabanski "Cade" <cade@biscom.net> <cade@datamax.bg> <cade@cpan.org> http://cade.datamax.bg perl v5.18.2 2013-07-08 Time::Progress(3)
All times are GMT -4. The time now is 05:17 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy