Sponsored Content
Full Discussion: Parsing syslog from Linux
Top Forums Shell Programming and Scripting Parsing syslog from Linux Post 303037028 by RudiC on Sunday 21st of July 2019 03:33:03 AM
Old 07-21-2019
The code has to match NF fields against 9 items for every line; this will take its time, esp. on large files. I compared (timed) your code to mine on a medium sized sample data file and found that yours is roughly two to three times slower, so I don't understand the 27 min of my code vs. 6 min of your code. Still, going through my proposal again and trying to tease out a few percent, I came up with
Code:
awk '
BEGIN   {print HDLN = "eventtime|srcip|dstip|srcport|dstport|transip|transport|action|sessionid"
         MX = split (HDLN, HD, "|")
         for (i=1; i<=MX; i++) L[i] = length (HD[i]) + 1
        }
        {OUT = DL = ""
         for (i=1; i<=MX; i++)  {match ($0, HD[i] "=[^ ]*")
                                 OUT = OUT DL  substr ($0, RSTART + L[i], RLENGTH - L[i])
                                 DL = "|"
                                }
         print OUT 
        }
' file

Pls try and report back, esp. in comparison to your code in post #5 (don't forget you'll need to match the fields' sequence to the header's).
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need some help with parsing

I have a big xml file with little formatting in it. It contains over 600 messages that I need to break each message out in its own separate file. The xml file looks in the middle of it something like this: </Title></Msg><Msg><Opener> Hello how are you?<Title> Some says hello</Title><Body>... (3 Replies)
Discussion started by: quixoticking11
3 Replies

2. Shell Programming and Scripting

Perl parsing compared to Ksh parsing

#! /usr/local/bin/perl -w $ip = "$ARGV"; $rw = "$ARGV"; $snmpg = "/usr/local/bin/snmpbulkget -v2c -Cn1 -Cn2 -Os -c $rw"; $snmpw = "/usr/local/bin/snmpwalk -Os -c $rw"; $syst=`$snmpg $ip system sysName sysObjectID`; sysDescr.0 = STRING: Cisco Internetwork Operating System Software... (1 Reply)
Discussion started by: popeye
1 Replies

3. Shell Programming and Scripting

Parsing of file for Report Generation (String parsing and splitting)

Hey guys, I have this file generated by me... i want to create some HTML output from it. The problem is that i am really confused about how do I go about reading the file. The file is in the following format: TID1 Name1 ATime=xx AResult=yyy AExpected=yyy BTime=xx BResult=yyy... (8 Replies)
Discussion started by: umar.shaikh
8 Replies

4. Red Hat

Parsing a linux file and formatting it.

Hi, I have a linux file that has data like this.. REQUEST_ID|text^Ctext^Ctext^C REQUEST_ID|text^Ctext^C REQUEST_ID| REQUEST_ID| REQUEST_ID|text^Ctext^Ctext^Ctext^Ctext^Ctext^C.... Where ever I see a ^C character, I need to copy the corresponding REQUEST_ID and that part of the text to a new... (17 Replies)
Discussion started by: charithainfadev
17 Replies

5. Shell Programming and Scripting

Parsing kiwi syslog from Astaro

Hello, I am trying to parse this syslog pulling out and logging results to a file. The information I want is: scrip, scrport, dstip, dstport. I just want the numbers, not including the text part ie srcip=". Problem is, the column locations change, so I can't use the nice awk $1 $2 etc to... (4 Replies)
Discussion started by: rmelnik
4 Replies

6. UNIX for Dummies Questions & Answers

Parsing linux commands through FTP

Hi Techies, I have made a shell script which stores the output of it in a text file. then i wanted to fetch that text file using windows scheduler in my windows xp desktop which i did successfully using the below mentioned ftp .bat file : @echo off @echo ftp_user>ftp_test.scr @echo... (0 Replies)
Discussion started by: gemnian.g
0 Replies

7. Shell Programming and Scripting

Help - Parsing data in XML in Linux

Hi, I have an XML file in Linux and it contains a long string of characters. The last part of the file is like ....... ....... ....... CAD</MarketDescription></InvestorTransaction></AdvisorAccount></DivisionAdvisor></Division>... (3 Replies)
Discussion started by: naveed
3 Replies

8. Shell Programming and Scripting

Specific string parsing in Linux/UNIX

Hi, I have a string which can be completely unstructred. I am looking to parse out values within that String. Here is an example <Random Strings> String1=<some number a> String2=<some number b> String3=<some number c> Satish=<some number d> String4=<some number e> I only want to parse out... (1 Reply)
Discussion started by: satishrao
1 Replies

9. SuSE

Location and name of SYSLOG in SUSE Linux

Esteemed listers, Where is the location of SYSLOG file? In etc/auditd.conf script, the log_file location is '/var/log/audit/audit.log' as below. Is this the location where SYSLOG is stored? Thank you in advance, log_file = /var/log/audit/audit.log log_format = RAW... (3 Replies)
Discussion started by: JDBA
3 Replies

10. Programming

Openlog and syslog in red-hat Linux doesn't write any thing to /var/log/*

Using redhat 64 bit ver 6.2 I have simple c++ app that is trying to write to syslog like this: /* try to write massage into linux log */ void foo::writeToSyslog() { openlog("testlogfoo", 0, 24); // Send the message. ... (1 Reply)
Discussion started by: umen
1 Replies
Padre::Document::Perl::Beginner(3pm)			User Contributed Perl Documentation		      Padre::Document::Perl::Beginner(3pm)

NAME
Padre::Document::Perl::Beginner - naive implementation of some beginner specific error checking SYNOPSIS
use Padre::Document::Perl::Beginner; my $beginner = Padre::Document::Perl::Beginner->new; if (not $beginner->check($data)) { warn $beginner->error; } DESCRIPTION
This is a naive implementation. It needs to be replaced by one using PPI. In Perl 5 there are lots of pitfalls the unaware, especially the beginner can easily fall in. While some might expect the Perl compiler itself would catch those it does not (yet ?) do it. So we took the initiative and added a beginners mode to Padre in which these extra issues are checked. Some are real problems that would trigger an error anyway we just make them a special case with a more specific error message. (e.g. "use warning;" without the trailing s) Others are valid code that can be useful in the hands of a master but that are poisonous when written by mistake by someone who does not understand them. (e.g. "if ($x = /value/) { }" ). This module provides a method called "check" that can check a Perl script (provided as parameter as a single string) and recognize problematic code. Examples See <http://padre.perlide.org/ticket/52> and <http://www.perlmonks.org/?node_id=728569> Cases o split /,/, @data; Here @data is in scalar context returning the number of elements. Spotted in this form: split /,/, @ARGV; o use warning; s is missing at the end. o map { $_; } (@items),$extra_item; is the same as map { $_; } (@items,$extra_item); but you usually want (map { $_; } (@items)),$extra_item; which means: map all @items and them add $extra_item without mapping it. o Warn about Perl-standard package names being reused package DB; o $x = chomp $y; print chomp $y; o map { s/foo/bar/; } (@items); This returns an array containing true or false values (s/// - return value). Use map { s/foo/bar/; $_; } (@items); to actually change the array via s///. o <@X> o if ($x = /bla/) { } o Pipe | in open() not at the end or the beginning. o open($ph, "| something |"); o Regular expression starting with a quantifier such as /+.../ o } else if { o } elseif { o close; HOW TO ADD ANOTHER ONE
Please feel free to add as many checks as you like. This is done in three steps: Add the test Add one (or more) tests for this case to t/75-perl-beginner.t The test should be successful when your supplied sample fails the check and returns the correct error message. As texts of error messages may change, try to match a good part which allows identification of the message but don't match the very exact text. Tests could use either one-liners written as strings within the test file or external support files. There are samples for both ways in the test script. Add the check Add the check to the check-sub of this file (Document/Perl/Beginner.pm). There are plenty samples here. Remember to add a sample (and maybe short description) what would fail the test. Run the test script to match your test case(s) to the new check. Add the configuration option Go to Config.pm, look for the beginner error checks configuration and add a new setting for your new check there. It defaults to 1 (run the check), but a user could turn it off by setting this to 0 within the Padre configuration file. COPYRIGHT &; LICENSE Copyright 2008-2012 The Padre development team as listed in Padre.pm. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of the license can be found in the LICENSE file included with this module. perl v5.14.2 2012-06-27 Padre::Document::Perl::Beginner(3pm)
All times are GMT -4. The time now is 09:41 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy