Sponsored Content
Full Discussion: Parsing syslog from Linux
Top Forums Shell Programming and Scripting Parsing syslog from Linux Post 303037068 by Chubler_XL on Monday 22nd of July 2019 06:40:00 PM
Old 07-22-2019
You could try using index and substr instead of match to avoid regex overheads. This takes about 2mins for a 2GB file on my system:

Code:
awk '
BEGIN   {
    HDLN = "eventtime|srcip|dstip|srcport|dstport|transip|transport|" \
           "action|sessionid"
    MX = split (HDLN, HD, "|")
    print HDLN
}
{
  DL = ""
  for (i=1; i<=MX; i++)  {
      s=index($0, HD[i] "=")
      if(s) {
          s += length(HD[i]) + 1
          e=index(substr($0,s)," ")-1
          printf DL substr($0, s, e)
      } else printf DL
      DL = "|" 
  }
  printf "\n"
}' infile

--- Post updated at 09:40 AM ---

As a further test I used the above logic in C, and it finished in 1min 20sec on my system. This has to be close to the fastest you could expect:

Code:
#include <stdio.h>
#include <string.h>

int main()
{
   char line_buff[1024];
   int i;
   char *s;
   char dl[2] = "";
   char *match[] = {
     "eventtime=",
     "srcip=",
     "dstip=",
     "srcport=",
     "dstport=",
     "transip=",
     "transport=",
     "action=",
     "sessionid=",
      NULL };


   printf("%.*s", strlen(match[0])-1, match[0]);
   for(i=1;match[i];i++) printf("|%.*s", strlen(match[i])-1, match[i]);
   printf("\n");

   while (!feof(stdin)) {
       if (fgets(line_buff, 1024, stdin)) {
           dl[0]='\0';
           for(i=0;match[i];i++) {
              s=strstr(line_buff, match[i]);
              if(s) {
                printf("%s", dl);
                s+=strlen(match[i]);
                while(*s && *s!=' ') printf("%c", *(s++));
              } else printf("%s", dl);
              strcpy(dl, "|");
            }
           printf("\n");
       }
   }
   return 0;
}


Last edited by Chubler_XL; 07-22-2019 at 07:38 PM.. Reason: Fix indenting
This User Gave Thanks to Chubler_XL For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need some help with parsing

I have a big xml file with little formatting in it. It contains over 600 messages that I need to break each message out in its own separate file. The xml file looks in the middle of it something like this: </Title></Msg><Msg><Opener> Hello how are you?<Title> Some says hello</Title><Body>... (3 Replies)
Discussion started by: quixoticking11
3 Replies

2. Shell Programming and Scripting

Perl parsing compared to Ksh parsing

#! /usr/local/bin/perl -w $ip = "$ARGV"; $rw = "$ARGV"; $snmpg = "/usr/local/bin/snmpbulkget -v2c -Cn1 -Cn2 -Os -c $rw"; $snmpw = "/usr/local/bin/snmpwalk -Os -c $rw"; $syst=`$snmpg $ip system sysName sysObjectID`; sysDescr.0 = STRING: Cisco Internetwork Operating System Software... (1 Reply)
Discussion started by: popeye
1 Replies

3. Shell Programming and Scripting

Parsing of file for Report Generation (String parsing and splitting)

Hey guys, I have this file generated by me... i want to create some HTML output from it. The problem is that i am really confused about how do I go about reading the file. The file is in the following format: TID1 Name1 ATime=xx AResult=yyy AExpected=yyy BTime=xx BResult=yyy... (8 Replies)
Discussion started by: umar.shaikh
8 Replies

4. Red Hat

Parsing a linux file and formatting it.

Hi, I have a linux file that has data like this.. REQUEST_ID|text^Ctext^Ctext^C REQUEST_ID|text^Ctext^C REQUEST_ID| REQUEST_ID| REQUEST_ID|text^Ctext^Ctext^Ctext^Ctext^Ctext^C.... Where ever I see a ^C character, I need to copy the corresponding REQUEST_ID and that part of the text to a new... (17 Replies)
Discussion started by: charithainfadev
17 Replies

5. Shell Programming and Scripting

Parsing kiwi syslog from Astaro

Hello, I am trying to parse this syslog pulling out and logging results to a file. The information I want is: scrip, scrport, dstip, dstport. I just want the numbers, not including the text part ie srcip=". Problem is, the column locations change, so I can't use the nice awk $1 $2 etc to... (4 Replies)
Discussion started by: rmelnik
4 Replies

6. UNIX for Dummies Questions & Answers

Parsing linux commands through FTP

Hi Techies, I have made a shell script which stores the output of it in a text file. then i wanted to fetch that text file using windows scheduler in my windows xp desktop which i did successfully using the below mentioned ftp .bat file : @echo off @echo ftp_user>ftp_test.scr @echo... (0 Replies)
Discussion started by: gemnian.g
0 Replies

7. Shell Programming and Scripting

Help - Parsing data in XML in Linux

Hi, I have an XML file in Linux and it contains a long string of characters. The last part of the file is like ....... ....... ....... CAD</MarketDescription></InvestorTransaction></AdvisorAccount></DivisionAdvisor></Division>... (3 Replies)
Discussion started by: naveed
3 Replies

8. Shell Programming and Scripting

Specific string parsing in Linux/UNIX

Hi, I have a string which can be completely unstructred. I am looking to parse out values within that String. Here is an example <Random Strings> String1=<some number a> String2=<some number b> String3=<some number c> Satish=<some number d> String4=<some number e> I only want to parse out... (1 Reply)
Discussion started by: satishrao
1 Replies

9. SuSE

Location and name of SYSLOG in SUSE Linux

Esteemed listers, Where is the location of SYSLOG file? In etc/auditd.conf script, the log_file location is '/var/log/audit/audit.log' as below. Is this the location where SYSLOG is stored? Thank you in advance, log_file = /var/log/audit/audit.log log_format = RAW... (3 Replies)
Discussion started by: JDBA
3 Replies

10. Programming

Openlog and syslog in red-hat Linux doesn't write any thing to /var/log/*

Using redhat 64 bit ver 6.2 I have simple c++ app that is trying to write to syslog like this: /* try to write massage into linux log */ void foo::writeToSyslog() { openlog("testlogfoo", 0, 24); // Send the message. ... (1 Reply)
Discussion started by: umen
1 Replies
SYSLOG(8)							  System Logging							 SYSLOG(8)

NAME
syslog-ng, syslogd DESCRIPTION
There are different syslog daemon implementations supported as the system's syslog service, currently syslogd, syslog-ng and rsyslogd The first installed daemon activates itself for the syslog service. Starting with openSUSE-11.2, it is rsyslogd, before it was syslog-ng. But this depends on the software selection during the installation. The name of the daemon used as syslog service is specified in the SYSLOG_DAEMON variable in /etc/sysconfig/syslog. The yast2 sysconfig module provides a comfortable way to switch to another installed daemon and restart the service. The /etc/init.d/syslog init script is able to handle all supported daemons. BUGS
Please report bugs at <http://www.suse.de/feedback> AUTHOR
Juergen Weigert <jw@novell.com> Marius Tomaschewski <mt@novell.com> SEE ALSO
sysklogd(8) syslogd(8) syslog.conf(5) syslog-ng(8) syslog-ng.conf(5) rsyslogd(8) rsyslog.conf(5) syslog May 2008 SYSLOG(8)
All times are GMT -4. The time now is 01:56 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy