Help to parse syslog with perl


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
# 1  
Help to parse syslog with perl

[QUOTE=arm;303037305]logver=56 idseq=63256900099118326 itime=1563205190 devid=FG-5KDTB18800138 devname=LAL-C1-FGT-03 vd=USER date=2019-07-15 time=18:39:49 logid="0000000013" type="traffic"
subtype="forward" level="notice" eventtime=1563205189 srcip=11.3.3.17 srcport=50544 srcintf="SGI-CORE.123" srcintfrole="undefined" dstip=12.0.1.1 dstport=443 dsti
ntf="FA-SPI.100" dstintfrole="undefined" poluuid="230d4d26-AAAA-51e9-b9d1-7bf4c828f000" sessionid=20639817 proto=6 action="server-rst" policyid=10 policytype="policy" s
ervice="HTTPS" dstcountry="United State" srccountry="Reserved" trandisp="snat" transip=11.1.1.1 transport=5092 duration=71 sentbyte=093 rcvdbyte=213 sentpkt=11 rcv
dpkt=16 appcat="unscanned"

I used below script to parsing 1000000 records
Code:
#!/usr/bin/env perl
use strict;
use warnings;
while( <> ) {
    if ( /^(?=.*eventtime=(\S+))(?=.*srcip=(\S+))(?=.*srcport=(\S+))(?=.*dstip=(\S+))(?=.*dstport=(\S+))(?=.*sessionid=(\S+))(?=.*action=(\S+))(?=.*policyid=(\S+))(?=.*service=(\S+))(?=.*dstcountry=(\S+))(?=.*transip=(\S+))(?=.*transport=(\S+))(?=.*duration=(\S+)).*$/ ) {
            print "$1|$2|$3|$4|$5|$6|$7|$8|$9|$10|$11|$12|$13\n";
                }
                }


the problem here is didn't manage to find the correct "regular expression" to match dstcountry , what I need is to give me "United State" not "United

1563205189|11.3.3.17|50544|12.0.1.1 |443|20585519|"server-rst"|10|"HTTPS"|"United|11.1.1.1|5092|71
# 2  
Try this:

Code:
#!/usr/bin/env perl
use strict;
use warnings;
while( <> ) {
    if ( /^(?=.*eventtime=(\S+))(?=.*srcip=(\S+))(?=.*srcport=(\S+))
           (?=.*dstip=(\S+))(?=.*dstport=(\S+))(?=.*sessionid=(\S+))
           (?=.*action=(\S+))(?=.*policyid=(\S+))(?=.*service=(\S+))
           (?=.*dstcountry=("[^"]+"|\S+))(?=.*transip=(\S+))
           (?=.*transport=(\S+))(?=.*duration=(\S+)).*$/x ) {
            print "$1|$2|$3|$4|$5|$6|$7|$8|$9|$10|$11|$12|$13\n";
                }
                }

--- Post updated at 07:17 AM ---

Or if the quote are always there you can drop the |\S+ and if you want to strip the quotes use (?=.*dstcountry="([^"]+)")

Last edited by Chubler_XL; 07-29-2019 at 05:11 PM.. Reason: wrap long line
This User Gave Thanks to Chubler_XL For This Post:
# 3  
@Chubler_XL
billion thanks for prompt response , in fact I have no "perl" background so if you can do me a favor and answer my below inquires it would be highly appreciated:

1. Could you please tell me where can I learn such regular expressions , any link or any book should go though ?
2. What is the equivalent regex for :
Code:
(?=.*dstcountry=("[^"]+"|\S+))  and  (?=.*dstcountry="([^"]+)")

if I wanna do it using stream editor in bash:
Code:
sed 's/?/?/g'

# 4  
Many modern sed implementations support extended regular expressions (POSIX ERE) via the -E option . This is not a powerful as perl and does not support named capture groups, but should be sufficient for your requirement.

Using ERE with the sed -n option (to suppress automatic printing of pattern space) we can do:

Code:
sed -En 's/.*dstcountry=("[^"]+"|\S+).*/\1/p'
sed -En 's/.*dstcountry="([^"]+)".*/\1/p'

\1 in the replace part of the substitution is referring to the first capture group the same as $1 in perl.

Sorry I don't know of a good source to learn all this stuff, I just picked it up over the years of using these products. Perhaps try searching for "learning regex" or "regular expression examples" using your favorite search engine.

I did a quick search myself and came across this quite nice regex cheat sheet, it seems to cover a lot of features and is fairly easy to use Regex Cheat Sheet
These 2 Users Gave Thanks to Chubler_XL For This Post:
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl to parse a variety of formats

The below perl script parses a variety of formats. If I use the numeric text file as input the script works correctly. However using the alpha text file as input there is a black output file. The portion in bold splits the field to parse f or NC_000023.10:g.153297761C>A into a variable $common but... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

Perl to parse

The below code works great to parse out a file if the input is in the attached SNP format ">". perl -ne 'next if $.==1; while(/\t*NC_(\d+)\.\S+g\.(\d+)()>()/g){printf("%d\t%d\t%d\t%s\t%s\n",$1,$2,$2,$3,$4,$5)}' out_position.txt > out_parse.txt My question is if there is another format in... (10 Replies)
Discussion started by: cmccabe
10 Replies

3. Shell Programming and Scripting

awk or perl to parse file

I have an input file attached that I am trying to parse in tab-delimanted format: The chromosomal variant column contains all the information: parse rules: 1. 4 zeros after the NC_ and the digits before the . 2. digits after the g. repeated twice separated by a tab 3. letter before the > 4.... (10 Replies)
Discussion started by: cmccabe
10 Replies

4. Shell Programming and Scripting

Perl :: to parse the data from a string.

Hi folks, I have a line in log from which I need to parse few data. Jul 6 00:05:58 dg01aipagnfe01p %FWSM-3-106011: Deny inbound (No xlate) From the above... I need to parse the %FWSM-3-106011: substring. Another example Jul 13 00:08:55 dq01aipaynas01p %FWSM-6-302010: 2 in use, 1661... (3 Replies)
Discussion started by: scriptscript
3 Replies

5. Programming

Perl parse string

Hi Perl Guys I have another perl question I have the following code that i have written Getopt::Long::config(qw( permute bundling )); my $OPT = {}; GetOptions($OPT, qw( ver=s help|h )) or die "options parsing failed"; This will allow the user to do something like... (4 Replies)
Discussion started by: ab52
4 Replies

6. Shell Programming and Scripting

Perl parse error

Hello there, I em executing the following command in a perl script to append "\0" to the end of every line in a file: ###command start my $cmd = qx{"C:\\gawk" '{print $0 "\\\0"}' C:\file.txt > C:\file_1.txt}; ###command end But i get the following error: ###error meaasge start... (2 Replies)
Discussion started by: nmattam
2 Replies

7. Shell Programming and Scripting

perl parse log

Hi anyone can help.how can i get all second column data in this log below?? x 799002577959.pdf, 25728 bytes, 51 tape blocks x 800002357216.pdf, 25728 bytes, 51 tape blocks x aadb090910.txt, 80424 bytes, 158 tape blocks x tsese090909.txt, 13974 bytes, 28 tape blocks (4 Replies)
Discussion started by: netxus
4 Replies

8. Shell Programming and Scripting

Perl Parse

Hi I'm writing simple perl script to parse the ftp log as below: Local directory now /home/user/testing 227 Entering Passive Mode (192,254,19,34,8,228). 125 Data connection already open; Transfer starting. 09-25-09 02:33PM 25333629 abc.tar 09-14-09 12:50PM 18015752... (1 Reply)
Discussion started by: netxus
1 Replies

9. Shell Programming and Scripting

perl parse line

Dear all anyone willling to help me..i have try so many time but still failed to get the ip address for line when i print the line is like below Connected to 192.168.1.13 #!/usr/local/bin/perl foreach $line(@lines){ if ($line =~ /connected to/) { $line=~/connected to(.*?) /; ... (2 Replies)
Discussion started by: netxus
2 Replies

10. Shell Programming and Scripting

CSV File parse help in Perl

Folks, I have a bit of an issue trying to obtain some data from a csv file using PERL. I can sort the file and remove any duplicates leaving only 4 or 5 rows containing data. My problem is that the data contained in the original file contains a lot more columns and when I try ro run this script... (13 Replies)
Discussion started by: lodey
13 Replies

Featured Tech Videos