Parsing syslog from Linux


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parsing syslog from Linux
# 8  
Old 07-21-2019
The code has to match NF fields against 9 items for every line; this will take its time, esp. on large files. I compared (timed) your code to mine on a medium sized sample data file and found that yours is roughly two to three times slower, so I don't understand the 27 min of my code vs. 6 min of your code. Still, going through my proposal again and trying to tease out a few percent, I came up with
Code:
awk '
BEGIN   {print HDLN = "eventtime|srcip|dstip|srcport|dstport|transip|transport|action|sessionid"
         MX = split (HDLN, HD, "|")
         for (i=1; i<=MX; i++) L[i] = length (HD[i]) + 1
        }
        {OUT = DL = ""
         for (i=1; i<=MX; i++)  {match ($0, HD[i] "=[^ ]*")
                                 OUT = OUT DL  substr ($0, RSTART + L[i], RLENGTH - L[i])
                                 DL = "|"
                                }
         print OUT 
        }
' file

Pls try and report back, esp. in comparison to your code in post #5 (don't forget you'll need to match the fields' sequence to the header's).
# 9  
Old 07-21-2019
Hi, also try something like this:

Code:
awk '
  NR==FNR {
    split($0,Label,"|")
    next
  }
  {
    for(i=1; i<=NF; i++) {
      split($i,F,"=")
      gsub(/"/,x,F[2])
      Key[F[1]]=F[2]
    }
    $0=x
    for(i in Label) 
      $i=Key[Label[i]]
    print
  }
' OFS=\| file2 file1

Where file2 contains:
Code:
eventtime|srcip|dstip|srcport|dstport|transip|transport|action|sessionid

Output:
Code:
1563205189|11.3.3.17|12.0.1.1|50544|443|11.1.1.1|5092|server-rst|20639817

# 10  
Old 07-21-2019
I did some testing and RudiC's second method turned out to be fastest.

I would suggest trying mawk

In tests I conducted with RudiC's approach mawk was several orders faster than regular awk or gawk..

Last edited by Scrutinizer; 07-21-2019 at 01:30 PM..
This User Gave Thanks to Scrutinizer For This Post:
# 11  
Old 07-21-2019
thanks but it does not work for me
# 12  
Old 07-21-2019
What does not work?
# 13  
Old 07-22-2019
You could try using index and substr instead of match to avoid regex overheads. This takes about 2mins for a 2GB file on my system:

Code:
awk '
BEGIN   {
    HDLN = "eventtime|srcip|dstip|srcport|dstport|transip|transport|" \
           "action|sessionid"
    MX = split (HDLN, HD, "|")
    print HDLN
}
{
  DL = ""
  for (i=1; i<=MX; i++)  {
      s=index($0, HD[i] "=")
      if(s) {
          s += length(HD[i]) + 1
          e=index(substr($0,s)," ")-1
          printf DL substr($0, s, e)
      } else printf DL
      DL = "|" 
  }
  printf "\n"
}' infile

--- Post updated at 09:40 AM ---

As a further test I used the above logic in C, and it finished in 1min 20sec on my system. This has to be close to the fastest you could expect:

Code:
#include <stdio.h>
#include <string.h>

int main()
{
   char line_buff[1024];
   int i;
   char *s;
   char dl[2] = "";
   char *match[] = {
     "eventtime=",
     "srcip=",
     "dstip=",
     "srcport=",
     "dstport=",
     "transip=",
     "transport=",
     "action=",
     "sessionid=",
      NULL };


   printf("%.*s", strlen(match[0])-1, match[0]);
   for(i=1;match[i];i++) printf("|%.*s", strlen(match[i])-1, match[i]);
   printf("\n");

   while (!feof(stdin)) {
       if (fgets(line_buff, 1024, stdin)) {
           dl[0]='\0';
           for(i=0;match[i];i++) {
              s=strstr(line_buff, match[i]);
              if(s) {
                printf("%s", dl);
                s+=strlen(match[i]);
                while(*s && *s!=' ') printf("%c", *(s++));
              } else printf("%s", dl);
              strcpy(dl, "|");
            }
           printf("\n");
       }
   }
   return 0;
}


Last edited by Chubler_XL; 07-22-2019 at 07:38 PM.. Reason: Fix indenting
This User Gave Thanks to Chubler_XL For This Post:
# 14  
Old 07-26-2019
thanks guys but now the log gets bigger and some other fields are added some others are deleted , what I need now is to help me to

Quote:
Logsample
Code:
logver=56 idseq=63256900099118326 itime=1563205190 devid=FG-5KDTB18800138 devname=LAL-C1-FGT-03 vd=USER date=2019-07-15 time=18:39:49 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1563205189 srcip=11.3.3.17 srcport=50544 srcintf="SGI-CORE.123" srcintfrole="undefined" dstip=12.0.1.1 dstport=443 dstintf="FA-SPI.100" dstintfrole="undefined" poluuid="230d4d26-AAAA-51e9-b9d1-7bf4c828f000" sessionid=20639817 proto=6 action="server-rst" policyid=10 policytype="policy" service="HTTPS" dstcountry="Germany" srccountry="Reserved" trandisp="snat" transip=11.1.1.1 transport=5092 duration=71 sentbyte=093 rcvdbyte=213 sentpkt=11 rcvdpkt=16 appcat="unscanned"

logver=56 idseq=63256900099118326 itime=1563205190 devid=FG-5KDTB18800138 devname=LAL-C1-FGT-03 vd=USER date=2019-07-15 time=18:39:49 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1563205190 srcip=11.3.3.17 srcport=50544 srcintf="SGI-CORE.123" srcintfrole="undefined" dstip=13.0.1.1 dstport=80 dstintf="FA-SPI.100" dstintfrole="undefined" poluuid="230d4d26-AAAA-51e9-b9d1-7bf4c828f000" sessionid=20639824 proto=6 action="close" policyid=34 policytype="policy" service="UDP" dstcountry="United State" srccountry="Reserved" trandisp="snat" transip=14.1.1.1 transport=5092 duration=50 sentbyte=093 rcvdbyte=213 sentpkt=11 rcvdpkt=16 appcat="unscanned"


logver=56 idseq=63256900099118326 itime=1563205190 devid=FG-5KDTB18800138 devname=LAL-C1-FGT-03 vd=USER date=2019-07-15 time=18:39:49 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1563205100 srcip=11.3.3.17 srcport=50590 srcintf="SGI-CORE.123" srcintfrole="undefined" dstip=1.0.1.1 dstport=80 dstintf="FA-SPI.100" dstintfrole="undefined" poluuid="230d4d26-AAAA-51e9-b9d1-7bf4c828f000" sessionid=20639817 proto=6  policyid=34 policytype="policy" service="UDP/10"  srccountry="Reserved" trandisp="snat"  duration=60 sentbyte=093 rcvdbyte=213 sentpkt=11 rcvdpkt=16 appcat="unscanned"



eventtime|srcip|dstip|srcport|dstport|transip|transport|action|sessionid|service|policyid|dstcountry|duration    --> header no need to be shown on output and if any field was missing then it will left as empty between pipe delimiter as  the case at line 3 ||   

1563205189|11.3.3.17|12.0.1.1|50544|443|11.1.1.1|5092|server-rst|20639817|HTTPS|10|Germany|71
1563205190|11.3.3.17|13.0.1.1|50544|80|14.1.1.1|5092|closet|20639824|UDP|34|United State|50
1563205100|11.3.3.17|1.0.1.1|50590|80||||20639817|UDP/10|34||60


Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Openlog and syslog in red-hat Linux doesn't write any thing to /var/log/*

Using redhat 64 bit ver 6.2 I have simple c++ app that is trying to write to syslog like this: /* try to write massage into linux log */ void foo::writeToSyslog() { openlog("testlogfoo", 0, 24); // Send the message. ... (1 Reply)
Discussion started by: umen
1 Replies

2. SuSE

Location and name of SYSLOG in SUSE Linux

Esteemed listers, Where is the location of SYSLOG file? In etc/auditd.conf script, the log_file location is '/var/log/audit/audit.log' as below. Is this the location where SYSLOG is stored? Thank you in advance, log_file = /var/log/audit/audit.log log_format = RAW... (3 Replies)
Discussion started by: JDBA
3 Replies

3. Shell Programming and Scripting

Specific string parsing in Linux/UNIX

Hi, I have a string which can be completely unstructred. I am looking to parse out values within that String. Here is an example <Random Strings> String1=<some number a> String2=<some number b> String3=<some number c> Satish=<some number d> String4=<some number e> I only want to parse out... (1 Reply)
Discussion started by: satishrao
1 Replies

4. Shell Programming and Scripting

Help - Parsing data in XML in Linux

Hi, I have an XML file in Linux and it contains a long string of characters. The last part of the file is like ....... ....... ....... CAD</MarketDescription></InvestorTransaction></AdvisorAccount></DivisionAdvisor></Division>... (3 Replies)
Discussion started by: naveed
3 Replies

5. UNIX for Dummies Questions & Answers

Parsing linux commands through FTP

Hi Techies, I have made a shell script which stores the output of it in a text file. then i wanted to fetch that text file using windows scheduler in my windows xp desktop which i did successfully using the below mentioned ftp .bat file : @echo off @echo ftp_user>ftp_test.scr @echo... (0 Replies)
Discussion started by: gemnian.g
0 Replies

6. Shell Programming and Scripting

Parsing kiwi syslog from Astaro

Hello, I am trying to parse this syslog pulling out and logging results to a file. The information I want is: scrip, scrport, dstip, dstport. I just want the numbers, not including the text part ie srcip=". Problem is, the column locations change, so I can't use the nice awk $1 $2 etc to... (4 Replies)
Discussion started by: rmelnik
4 Replies

7. Red Hat

Parsing a linux file and formatting it.

Hi, I have a linux file that has data like this.. REQUEST_ID|text^Ctext^Ctext^C REQUEST_ID|text^Ctext^C REQUEST_ID| REQUEST_ID| REQUEST_ID|text^Ctext^Ctext^Ctext^Ctext^Ctext^C.... Where ever I see a ^C character, I need to copy the corresponding REQUEST_ID and that part of the text to a new... (17 Replies)
Discussion started by: charithainfadev
17 Replies

8. Shell Programming and Scripting

Parsing of file for Report Generation (String parsing and splitting)

Hey guys, I have this file generated by me... i want to create some HTML output from it. The problem is that i am really confused about how do I go about reading the file. The file is in the following format: TID1 Name1 ATime=xx AResult=yyy AExpected=yyy BTime=xx BResult=yyy... (8 Replies)
Discussion started by: umar.shaikh
8 Replies

9. Shell Programming and Scripting

Perl parsing compared to Ksh parsing

#! /usr/local/bin/perl -w $ip = "$ARGV"; $rw = "$ARGV"; $snmpg = "/usr/local/bin/snmpbulkget -v2c -Cn1 -Cn2 -Os -c $rw"; $snmpw = "/usr/local/bin/snmpwalk -Os -c $rw"; $syst=`$snmpg $ip system sysName sysObjectID`; sysDescr.0 = STRING: Cisco Internetwork Operating System Software... (1 Reply)
Discussion started by: popeye
1 Replies

10. Shell Programming and Scripting

Need some help with parsing

I have a big xml file with little formatting in it. It contains over 600 messages that I need to break each message out in its own separate file. The xml file looks in the middle of it something like this: </Title></Msg><Msg><Opener> Hello how are you?<Title> Some says hello</Title><Body>... (3 Replies)
Discussion started by: quixoticking11
3 Replies
Login or Register to Ask a Question