Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help to parse syslog with perl

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help to parse syslog with perl
# 1  
Old 07-29-2019
Help to parse syslog with perl
[QUOTE=arm;303037305]logver=56 idseq=63256900099118326 itime=1563205190 devid=FG-5KDTB18800138 devname=LAL-C1-FGT-03 vd=USER date=2019-07-15 time=18:39:49 logid="0000000013" type="traffic"
subtype="forward" level="notice" eventtime=1563205189 srcip=11.3.3.17 srcport=50544 srcintf="SGI-CORE.123" srcintfrole="undefined" dstip=12.0.1.1 dstport=443 dsti
ntf="FA-SPI.100" dstintfrole="undefined" poluuid="230d4d26-AAAA-51e9-b9d1-7bf4c828f000" sessionid=20639817 proto=6 action="server-rst" policyid=10 policytype="policy" s
ervice="HTTPS" dstcountry="United State" srccountry="Reserved" trandisp="snat" transip=11.1.1.1 transport=5092 duration=71 sentbyte=093 rcvdbyte=213 sentpkt=11 rcv
dpkt=16 appcat="unscanned"

I used below script to parsing 1000000 records
Code:
#!/usr/bin/env perl
use strict;
use warnings;
while( <> ) {
    if ( /^(?=.*eventtime=(\S+))(?=.*srcip=(\S+))(?=.*srcport=(\S+))(?=.*dstip=(\S+))(?=.*dstport=(\S+))(?=.*sessionid=(\S+))(?=.*action=(\S+))(?=.*policyid=(\S+))(?=.*service=(\S+))(?=.*dstcountry=(\S+))(?=.*transip=(\S+))(?=.*transport=(\S+))(?=.*duration=(\S+)).*$/ ) {
            print "$1|$2|$3|$4|$5|$6|$7|$8|$9|$10|$11|$12|$13\n";
                }
                }


the problem here is didn't manage to find the correct "regular expression" to match dstcountry , what I need is to give me "United State" not "United

1563205189|11.3.3.17|50544|12.0.1.1 |443|20585519|"server-rst"|10|"HTTPS"|"United|11.1.1.1|5092|71
# 2  
Old 07-29-2019
Try this:

Code:
#!/usr/bin/env perl
use strict;
use warnings;
while( <> ) {
    if ( /^(?=.*eventtime=(\S+))(?=.*srcip=(\S+))(?=.*srcport=(\S+))
           (?=.*dstip=(\S+))(?=.*dstport=(\S+))(?=.*sessionid=(\S+))
           (?=.*action=(\S+))(?=.*policyid=(\S+))(?=.*service=(\S+))
           (?=.*dstcountry=("[^"]+"|\S+))(?=.*transip=(\S+))
           (?=.*transport=(\S+))(?=.*duration=(\S+)).*$/x ) {
            print "$1|$2|$3|$4|$5|$6|$7|$8|$9|$10|$11|$12|$13\n";
                }
                }

--- Post updated at 07:17 AM ---

Or if the quote are always there you can drop the |\S+ and if you want to strip the quotes use (?=.*dstcountry="([^"]+)")
Last edited by Chubler_XL; 07-29-2019 at 05:11 PM.. Reason: wrap long line
This User Gave Thanks to Chubler_XL For This Post:
arm
# 3  
Old 07-29-2019
@Chubler_XL
billion thanks for prompt response , in fact I have no "perl" background so if you can do me a favor and answer my below inquires it would be highly appreciated:

1. Could you please tell me where can I learn such regular expressions , any link or any book should go though ?
2. What is the equivalent regex for :
Code:
(?=.*dstcountry=("[^"]+"|\S+))  and  (?=.*dstcountry="([^"]+)")

if I wanna do it using stream editor in bash:
Code:
sed 's/?/?/g'

# 4  
Old 07-29-2019
Many modern sed implementations support extended regular expressions (POSIX ERE) via the -E option . This is not a powerful as perl and does not support named capture groups, but should be sufficient for your requirement.

Using ERE with the sed -n option (to suppress automatic printing of pattern space) we can do:

Code:
sed -En 's/.*dstcountry=("[^"]+"|\S+).*/\1/p'
sed -En 's/.*dstcountry="([^"]+)".*/\1/p'

\1 in the replace part of the substitution is referring to the first capture group the same as $1 in perl.

Sorry I don't know of a good source to learn all this stuff, I just picked it up over the years of using these products. Perhaps try searching for "learning regex" or "regular expression examples" using your favorite search engine.

I did a quick search myself and came across this quite nice regex cheat sheet, it seems to cover a lot of features and is fairly easy to use Regex Cheat Sheet
These 2 Users Gave Thanks to Chubler_XL For This Post:
arm drl
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl to parse

The below code works great to parse out a file if the input is in the attached SNP format ">". perl -ne 'next if $.==1; while(/\t*NC_(\d+)\.\S+g\.(\d+)()>()/g){printf("%d\t%d\t%d\t%s\t%s\n",$1,$2,$2,$3,$4,$5)}' out_position.txt > out_parse.txt My question is if there is another format in... (10 Replies)
Discussion started by: cmccabe
10 Replies

2. Shell Programming and Scripting

Perl :: to parse the data from a string.

Hi folks, I have a line in log from which I need to parse few data. Jul 6 00:05:58 dg01aipagnfe01p %FWSM-3-106011: Deny inbound (No xlate) From the above... I need to parse the %FWSM-3-106011: substring. Another example Jul 13 00:08:55 dq01aipaynas01p %FWSM-6-302010: 2 in use, 1661... (3 Replies)
Discussion started by: scriptscript
3 Replies

3. Programming

Perl parse string

Hi Perl Guys I have another perl question I have the following code that i have written Getopt::Long::config(qw( permute bundling )); my $OPT = {}; GetOptions($OPT, qw( ver=s help|h )) or die "options parsing failed"; This will allow the user to do something like... (4 Replies)
Discussion started by: ab52
4 Replies

4. Shell Programming and Scripting

Perl parse error

Hello there, I em executing the following command in a perl script to append "\0" to the end of every line in a file: ###command start my $cmd = qx{"C:\\gawk" '{print $0 "\\\0"}' C:\file.txt > C:\file_1.txt}; ###command end But i get the following error: ###error meaasge start... (2 Replies)
Discussion started by: nmattam
2 Replies

5. Shell Programming and Scripting

Parse file contents in perl...

Hi, I have the file like this: #Contents of file 1 are: Dec 10 12:33:44 User1 Interface: Probe Dec 10 12:33:47 uSER1 SOME DATA Dec 10 12:33:47 user1 Interface: MSGETYPE Dec 10 12:34:48 user1 ID: 10. Dec 10 12:33:55 user1 Interface: MSGTYPE Dec 10 12:33:55 user1 Id: 9 ... (1 Reply)
Discussion started by: vanitham
1 Replies

6. Shell Programming and Scripting

perl parse log

Hi anyone can help.how can i get all second column data in this log below?? x 799002577959.pdf, 25728 bytes, 51 tape blocks x 800002357216.pdf, 25728 bytes, 51 tape blocks x aadb090910.txt, 80424 bytes, 158 tape blocks x tsese090909.txt, 13974 bytes, 28 tape blocks (4 Replies)
Discussion started by: netxus
4 Replies

7. Shell Programming and Scripting

Perl Parse

Hi I'm writing simple perl script to parse the ftp log as below: Local directory now /home/user/testing 227 Entering Passive Mode (192,254,19,34,8,228). 125 Data connection already open; Transfer starting. 09-25-09 02:33PM 25333629 abc.tar 09-14-09 12:50PM 18015752... (1 Reply)
Discussion started by: netxus
1 Replies

8. Shell Programming and Scripting

perl parse line

Dear all anyone willling to help me..i have try so many time but still failed to get the ip address for line when i print the line is like below Connected to 192.168.1.13 #!/usr/local/bin/perl foreach $line(@lines){ if ($line =~ /connected to/) { $line=~/connected to(.*?) /; ... (2 Replies)
Discussion started by: netxus
2 Replies

9. Shell Programming and Scripting

Perl Parse Word Cksum help

Hi all, I'm attempting to parse through a .bin file word by word and perform a cksum on each word using perl. I'm new to perl so I dont exactly know how to get started. Any help would be greatly appreciated. Thanks! (1 Reply)
Discussion started by: TeamUSA
1 Replies

10. Shell Programming and Scripting

CSV File parse help in Perl

Folks, I have a bit of an issue trying to obtain some data from a csv file using PERL. I can sort the file and remove any duplicates leaving only 4 or 5 rows containing data. My problem is that the data contained in the original file contains a lot more columns and when I try ro run this script... (13 Replies)
Discussion started by: lodey
13 Replies
Login or Register to Ask a Question