Parsing syslog from Linux Post: 303037068

Sponsored Content

Top Forums Shell Programming and Scripting Parsing syslog from Linux Post 303037068 by Chubler_XL on Monday 22nd of July 2019 06:40:00 PM

07-22-2019

Moderator

You could try using index and substr instead of match to avoid regex overheads. This takes about 2mins for a 2GB file on my system:

Code:

awk '
BEGIN   {
    HDLN = "eventtime|srcip|dstip|srcport|dstport|transip|transport|" \
           "action|sessionid"
    MX = split (HDLN, HD, "|")
    print HDLN
}
{
  DL = ""
  for (i=1; i<=MX; i++)  {
      s=index($0, HD[i] "=")
      if(s) {
          s += length(HD[i]) + 1
          e=index(substr($0,s)," ")-1
          printf DL substr($0, s, e)
      } else printf DL
      DL = "|" 
  }
  printf "\n"
}' infile

--- Post updated at 09:40 AM ---

As a further test I used the above logic in C, and it finished in 1min 20sec on my system. This has to be close to the fastest you could expect:

Code:

#include <stdio.h>
#include <string.h>

int main()
{
   char line_buff[1024];
   int i;
   char *s;
   char dl[2] = "";
   char *match[] = {
     "eventtime=",
     "srcip=",
     "dstip=",
     "srcport=",
     "dstport=",
     "transip=",
     "transport=",
     "action=",
     "sessionid=",
      NULL };


   printf("%.*s", strlen(match[0])-1, match[0]);
   for(i=1;match[i];i++) printf("|%.*s", strlen(match[i])-1, match[i]);
   printf("\n");

   while (!feof(stdin)) {
       if (fgets(line_buff, 1024, stdin)) {
           dl[0]='\0';
           for(i=0;match[i];i++) {
              s=strstr(line_buff, match[i]);
              if(s) {
                printf("%s", dl);
                s+=strlen(match[i]);
                while(*s && *s!=' ') printf("%c", *(s++));
              } else printf("%s", dl);
              strcpy(dl, "|");
            }
           printf("\n");
       }
   }
   return 0;
}

Last edited by Chubler_XL; 07-22-2019 at 07:38 PM.. Reason: Fix indenting

This User Gave Thanks to Chubler_XL For This Post:

Chubler_XL

View Public Profile for Chubler_XL

Find all posts by Chubler_XL

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need some help with parsing

I have a big xml file with little formatting in it. It contains over 600 messages that I need to break each message out in its own separate file. The xml file looks in the middle of it something like this: </Title></Msg><Msg><Opener> Hello how are you?<Title> Some says hello</Title><Body>...

2. Shell Programming and Scripting

Perl parsing compared to Ksh parsing

#! /usr/local/bin/perl -w $ip = "$ARGV"; $rw = "$ARGV"; $snmpg = "/usr/local/bin/snmpbulkget -v2c -Cn1 -Cn2 -Os -c $rw"; $snmpw = "/usr/local/bin/snmpwalk -Os -c $rw"; $syst=`$snmpg $ip system sysName sysObjectID`; sysDescr.0 = STRING: Cisco Internetwork Operating System Software...

3. Shell Programming and Scripting

Parsing of file for Report Generation (String parsing and splitting)

Hey guys, I have this file generated by me... i want to create some HTML output from it. The problem is that i am really confused about how do I go about reading the file. The file is in the following format: TID1 Name1 ATime=xx AResult=yyy AExpected=yyy BTime=xx BResult=yyy...

4. Red Hat

Parsing a linux file and formatting it.

Hi, I have a linux file that has data like this.. REQUEST_ID|text^Ctext^Ctext^C REQUEST_ID|text^Ctext^C REQUEST_ID| REQUEST_ID| REQUEST_ID|text^Ctext^Ctext^Ctext^Ctext^Ctext^C.... Where ever I see a ^C character, I need to copy the corresponding REQUEST_ID and that part of the text to a new...

5. Shell Programming and Scripting

Parsing kiwi syslog from Astaro

Hello, I am trying to parse this syslog pulling out and logging results to a file. The information I want is: scrip, scrport, dstip, dstport. I just want the numbers, not including the text part ie srcip=". Problem is, the column locations change, so I can't use the nice awk $1 $2 etc to...

6. UNIX for Dummies Questions & Answers

Parsing linux commands through FTP

Hi Techies, I have made a shell script which stores the output of it in a text file. then i wanted to fetch that text file using windows scheduler in my windows xp desktop which i did successfully using the below mentioned ftp .bat file : @echo off @echo ftp_user>ftp_test.scr @echo...

7. Shell Programming and Scripting

Help - Parsing data in XML in Linux

Hi, I have an XML file in Linux and it contains a long string of characters. The last part of the file is like ....... ....... ....... CAD</MarketDescription></InvestorTransaction></AdvisorAccount></DivisionAdvisor></Division>...

8. Shell Programming and Scripting

Specific string parsing in Linux/UNIX

Hi, I have a string which can be completely unstructred. I am looking to parse out values within that String. Here is an example <Random Strings> String1=<some number a> String2=<some number b> String3=<some number c> Satish=<some number d> String4=<some number e> I only want to parse out...

9. SuSE

Location and name of SYSLOG in SUSE Linux

Esteemed listers, Where is the location of SYSLOG file? In etc/auditd.conf script, the log_file location is '/var/log/audit/audit.log' as below. Is this the location where SYSLOG is stored? Thank you in advance, log_file = /var/log/audit/audit.log log_format = RAW...

10. Programming

Openlog and syslog in red-hat Linux doesn't write any thing to /var/log/*

Using redhat 64 bit ver 6.2 I have simple c++ app that is trying to write to syslog like this: /* try to write massage into linux log */ void foo::writeToSyslog() { openlog("testlogfoo", 0, 24); // Send the message. ...

LEARN ABOUT OSF1

syslog.auth

syslog.auth(4)						     Kernel Interfaces Manual						    syslog.auth(4)

NAME

       syslog.auth -  authorization file for accepting remote syslog messages

SYNOPSIS

       # format:  Each fully qualified host name on a separate line hostname.domain_name

DESCRIPTION

       The /etc/syslog.auth file specifies which remote hosts are allowed to forward syslog messages to the local host.  For the sake of security,
       only messages coming from remote hosts listed in the local /etc/syslog.auth file will be logged by the syslogd daemon.

       If the /etc/syslog.auth file does not exist, then messages coming from any host will be accepted.

       Each remote host name should appear in a separate line in /etc/syslog.auth. A line started with the # character is considered as a  comment
       and is thus ignored.

       A  host	name  must be a complete domain name such as trout.zk3.dec.com. If a domain host name is given, it must either appear in the local
       /etc/hosts file or be able to be resolved by the local name server (BIND).

       Note that a host name can have at most as many characters as defined by the  MAXHOSTNAMELEN constant in <sys/param.h>, although	each  line
       in the /etc/syslog.auth file can have up to 512 characters.

       The /etc/syslog.auth file must be owned by root and has a permission of 0600.

       To invoke a new version of the /etc/syslog.auth file, run the following command (as the super user) to initialize the  syslogd daemon: kill
       -HUP `cat /var/run/syslog.pid`

EXAMPLES

       The  following  example	provides  a  typical  authorization  file:  #  format:	 Each  fully  qualified  host  name  on  a  separate  line
       c3poid.rvo.dec.com r2d2id.ckt.dec.com

FILES

       Location of the authorization file.

RELATED INFORMATION

       Commands: syslogd(8), syslog(1)

       System Administration delim off

																    syslog.auth(4)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need some help with parsing

Discussion started by: quixoticking11

2. Shell Programming and Scripting

Perl parsing compared to Ksh parsing

Discussion started by: popeye

3. Shell Programming and Scripting

Parsing of file for Report Generation (String parsing and splitting)

Discussion started by: umar.shaikh

4. Red Hat

Parsing a linux file and formatting it.

Discussion started by: charithainfadev