Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using perl to obtain stats instead of grep

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using perl to obtain stats instead of grep
# 1  
Old 04-21-2010
Using perl to obtain stats instead of grep
Evening all, i hope you guys can help !!

I'm working on a pass purchase system and would like to run a script every day from cron which looks at the previous day log files and produce some kind of stats for passes purchased / success / failures.

Issues.

I have around 25 passes and for each pass im running a grep for the SUBS_BOOK_ACC followed by an egrep for success and failure. Each one i put into a variable and print at the end. It works but i'm concerned im wasting resources on a system which is highly utilised.

Code:
grep SUBS_BOOK_ACC bkpass_log_2010042000000.audit | egrep `EPASS088|success` | wc -l

some sample data from the log file

2010-04-20_00:01:02:307,SUBS_BOOK_ACC,system@bkpass_1271_718059_59480,441234567899,EPASS088,,success,9999
2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS088,,success,6585
2010-04-20_00:01:24:335,SUBS_BOOK_ACC,system@bkpass_1271_718080_786699,441234567899,EPASSV09,,PAY payment failed,211

ideal output would be something like

Pass ID Success Failures

EPASSV09 7338 21
EPASS088 3852 11
EPASS013 452 55

I understand that i can use perl to pull the data and inspect the file line by line, instead of using multiple greps.

I've started a perl script but im having issues understand the syntax. I've input the file but im struggling with the pattern matching. Can someone help me understand how i can inspect each line and search for a sting, count and print.?

Can someone please help. i understand chomp is used to split the line and that i split the line at the , mark then i try the pattern match

Code:
#!/usr/bin/perl -w
use POSIX;
 
$LOGFILE = "/export/home/user/stats.test";
open (LOGFILE) or die ("Could not open log file.");
 
foreach $line (<LOGFILE>){
 
        chomp($_);
        @line=split(",",$_);
        @time=split(",",$line[0]);
        if (( $line[0] = m/SUBS_BOOK_ACC/ )){
                $results{"$time[0]:$time[1]"}[0]++;
        }
        $results{"$time[0]:$time[1]"}[1]++;
print($results);

In the above code im trying to find out how many pas booking were attempted.

If anyone can help i'd be eternally grateful.

Many thanks

Jeffers
# 2  
Old 04-21-2010
If you want a quick and dirty one-liner (not in the literal sense though), you could do something like this -

Code:
$ 
$ 
$ cat -n f3
     1    2010-04-20_00:01:02:307,SUBS_BOOK_ACC,system@bkpass_1271_718059_59480,441234567899,EPASS088,,success,9999
     2    2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS088,,success,6585
     3    2010-04-20_00:01:24:335,SUBS_BOOK_ACC,system@bkpass_1271_718080_786699,441234567899,EPASSV09,,PAY payment failed,211
     4    2010-04-20_00:01:24:335,SUBS_BOOK_ACC,system@bkpass_1271_718080_786699,441234567899,EPASSV09,,PAY payment failed,222
     5    2010-04-20_00:01:24:335,SUBS_BOOK_ACC,system@bkpass_1271_718080_786699,441234567899,EPASSV09,,success,8888
     6    2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS088,,PAY payment failed,2345
     7    2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS013,,success,6585
$ 
$ ##
$ perl -F, -ane 'if ($F[6] eq "success"){$s{$F[4]}++} elsif ($F[6] =~ m/failed/){$f{$F[4]}++}; $x{$F[4]}=1;
                 END {
                   printf("%10s %10s %10s\n","Pass ID","Successes","Failures");
                   foreach $i (keys %x) {printf("%10s %10s %10s\n", $i, $s{$i} // 0, $f{$i} // 0)}
                 }' f3
   Pass ID  Successes   Failures
  EPASS013          1          0
  EPASSV09          1          2
  EPASS088          2          1
$ 
$

tyler_durden
# 3  
Old 04-21-2010
late to the party
but here is an answer basically the same as Tyler's derived from your code:

Code:
#!/usr/bin/perl -w
use POSIX;

open (LOGFILE, $ARGV[0]) or die ("Could not open log file.");

foreach $line (<LOGFILE>){
        chomp($line);
        if (length($line) > 0) {
                @lineArray=split(",",$line);
                if ( $lineArray[1] =~ /SUBS_BOOK_ACC/ ) {
                        $key = $lineArray[4];
                        if (! defined $resultsHash{ $key }) {
                                $resultsHash{ $key } = [0,0];
                        }
                        my $temp = $resultsHash{ $key };
                        if ( $lineArray[6] =~ /success/ ) {
                                @$temp[0]++;
                        } elsif ( $lineArray[6] =~ /failed/ ) {
                                @$temp[1]++;
                        }
                        $resultsHash{ $key } = $temp;
                }
        }
}

foreach (sort keys %resultsHash) {
    $temp = $resultsHash{$_};
    print "$_ : @$temp\n";
}

# 4  
Old 04-22-2010
Thanks Turk,

thats worked brilliantly. thank you very much.

If i want to add an additional field called "already booked" would i be right in assuming the following:

Pass ID Successes Failures already booked
EPASS013 1 0 0
EPASSV09 1 2 1
EPASS088 7 2 2

I've tried adding the following extra 0 [0,0,0], and the extra elseif

Code:
foreach $line (<LOGFILE>){
        chomp($line);
        if (length($line) > 0) {
                @lineArray=split(",",$line);
                if ( $lineArray[1] =~ /SUB_OPB_BOOK_FACT/ ) {
                        $key = $lineArray[4];
                        if (! defined $resultsHash{ $key }) {
                                $resultsHash{ $key } = [0,0,0];  
                        }
                        my $temp = $resultsHash{ $key };
                        if ( $lineArray[6] =~ /success/ ) {
                                @$temp[0]++;
                        } elsif ( $lineArray[6] =~ /failed/ ) {
                                @$temp[1]++;
                        } elsif ( $lineArray[6] =~ /subscription already booked/ ) {
                                @$temp[2]++;
                        }
                        $resultsHash{ $key } = $temp;
                }
        }
}
foreach (sort keys %resultsHash) {
        $temp = $resultsHash{$_};
        print "$_ : @$temp\n"; }

Nothing appears in the output? would it be the print statement that requires amending?

Many thanks guys for all your help, its much appreciated.

Jeffers
# 5  
Old 04-22-2010
Quote:
Originally Posted by durden_tyler
$ perl -F, -ane 'if ($F[6] eq "success"){$s{$F[4]}++} elsif ($F[6] =~ m/failed/){$f{$F[4]}++}; $x{$F[4]}=1;
END {
printf("%10s %10s %10s\n","Pass ID","Successes","Failures");
foreach $i (keys %x) {printf("%10s %10s %10s\n", $i, $s{$i} // 0, $f{$i} // 0)}
}' f3
Pass ID Successes Failures
EPASS013 1 0
EPASSV09 1 2
EPASS088 2 1
$
$
[/code]

tyler_durden
above in bold should be :-

Code:
foreach $i (keys %x) {printf("%10s %10s %10s\n", $i, $s{$i} || 0, $f{$i} || 0)}

BR
SmilieSmilieSmilie
# 6  
Old 04-22-2010
Hi Jeffers,

I made the changes that you had listed, and it worked fine for me:

Code:
#!/usr/bin/perl -w
use POSIX;

open (LOGFILE, $ARGV[0]) or die ("Could not open log file.");

foreach $line (<LOGFILE>){
        chomp($line);
        if (length($line) > 0) {
                @lineArray=split(",",$line);
                if ( $lineArray[1] =~ /SUBS_BOOK_ACC/ ) {
                        $key = $lineArray[4];
                        if (! defined $resultsHash{ $key }) {
                                $resultsHash{ $key } = [0,0,0];
                        }
                        $temp = $resultsHash{ $key };
                        if ( $lineArray[6] =~ /success/ ) {
                                @$temp[0]++;
                        } elsif ( $lineArray[6] =~ /failed/ ) {
                                @$temp[1]++;
                        } elsif ( $lineArray[6] =~ /subscription already booked/ ) {
                                @$temp[2]++;
                        }
                        $resultsHash{ $key } = $temp;
                }
        }
}

foreach (sort keys %resultsHash) {
    $temp = $resultsHash{$_};
    print "$_ : @$temp\n";
}

sample data and output:

Code:
eturk-linux 10:20:31 (~)> cat data.dat
2010-04-20_00:01:02:307,SUBS_BOOK_ACC,system@bkpass_1271_718059_59480,441234567899,EPASS088,,success,9999
2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS088,,success,6585
2010-04-20_00:01:24:335,SUBS_BOOK_ACC,system@bkpass_1271_718080_786699,441234567899,EPASSV09,,PAY payment failed,211
2010-04-20_00:01:24:335,SUBS_BOOK_ACC,system@bkpass_1271_718080_786699,441234567899,EPASSV09,,PAY payment failed,222
2010-04-20_00:01:24:335,SUBS_BOOK_ACC,system@bkpass_1271_718080_786699,441234567899,EPASSV09,,success,8888
2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS088,,PAY payment failed,2345
2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS013,,success,6585
2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS013,,subscription already booked,6585
2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS013,,subscription already booked,6585
2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS013,,subscription already booked,6585
2010-04-20_00:01:18:139,SUBS_BOOK_ACC,system@bkpass_1271_718076_442346,441234567899,EPASS013,,subscription already booked,6585
eturk-linux 10:20:52 (~)> ./filter.pl data.dat
EPASS013 : 1 0 4
EPASS088 : 2 1 0
EPASSV09 : 1 2 0
eturk-linux 10:20:58 (~)>

# 7  
Old 04-22-2010
Quote:
Originally Posted by ahmad.diab
above in bold should be :-

Code:
foreach $i (keys %x) {printf("%10s %10s %10s\n", $i, $s{$i} || 0, $f{$i} || 0)}

BR
SmilieSmilieSmilie
Nope, it was meant to be "//" operator (truth and definedness) and not the "||" operator (OR).

See what happens here:

Code:
$
$
$ perl -le '%x=qw(10 0 20 0); $k=20; printf("\nKey   = %5s\nValue = %5s\n\n", $k, $x{$k} || -9)'
 
Key   =    20
Value =    -9
 
$
$ # oops!, -9 is *not* the value of the key 20.
$ # 0 is the value of the key 20.
$ # Let's try again...
$
$ perl -le '%x=qw(10 0 20 0); $k=20; printf("\nKey   = %5s\nValue = %5s\n\n", $k, $x{$k} // -9)'
 
Key   =    20
Value =     0
 
$
$ # Now that's better !!
$

Smilie

tyler_durden
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need help in solving to obtain desired print output using awk or perl or any commands, Please help!!

I have an file which have data in lines as follows ad, findline=24,an=54,ab=34,av=64,ab=7989,ab65=34,aj=323,ay=34,au=545,ad=5545 ab,abc,an10=23,an2=24,an31=32,findline=00,an33=23,an32=26,an40=45,ac23=5,ac=87,al=76,ad=26... (3 Replies)
Discussion started by: deepKrish
3 Replies

2. Shell Programming and Scripting

How to grep a pattern in perl?

hello Everyone i am a newbie. i have a file which contains the following E:\gtmproj\script\i486_nt\obj\check_geomtools.exe: o:\portsrc\spg\system_1\i486_nt\advapps\TK-2\objmt\winclockmtq.lib E:\gtmproj\script\i486_nt\obj\check_geomtools.exe:... (12 Replies)
Discussion started by: Rashid Khan
12 Replies

3. Shell Programming and Scripting

Grep in PERL

Hi, Can anybody let me know how this grep will work. The input and output is not known. Also can you give me the details of any link where i can find clearly about grep Thanks in advance (1 Reply)
Discussion started by: irudayaraj
1 Replies

4. Shell Programming and Scripting

GREP Issue in Perl

Im storing multiple functions in a varaible called $check... The variable check contains the following: a() b() c() ... ..etc now im checking individually which function is kept in which file using GREP if ( grep \$check \i, <FILE> ) The problem is im getting the output for the... (1 Reply)
Discussion started by: rajkrishna89
1 Replies

5. Shell Programming and Scripting

Perl + and Grep

Hi All i have this script that uses glob to look in /var/log/messages.* my @messagefiles = glob "/var/log/messages.*"; and the code that uses it is this grep { /NVRM: Xid/ } @messages) but this spits out this /var/log/messages-20111030:Oct 25 13:43:04 brent kernel: NVRM:... (10 Replies)
Discussion started by: ab52
10 Replies

6. Shell Programming and Scripting

grep in perl

Hello I want to grep a line from a file saved in some directory. Can anyone please correct the code below: #!/usr/bin/perl -w $file = "/home/output.txt" $grep_line = "closing zip for topic"; `grep $grep_line* $file`; (1 Reply)
Discussion started by: sureshcisco
1 Replies

7. Shell Programming and Scripting

How to execute Grep in Perl.

$ grep edge test_1 |sort|uniq >result.txt $more result.txt edge-a-pas01.com 10.12.10.11 edge-b-pas02.com 10.12.10.12 edge-c-pas03.com 10.12.10.50 edge-d-pas03.com 10.12.10.10 how do we execute the above grep command using perl? Thanks in advance. (3 Replies)
Discussion started by: sureshcisco
3 Replies

8. Shell Programming and Scripting

Perl grep

OK here's the situation: I have got these lines which I have got to parse. If the line contains a particular string and any element from a previously defined array I need to take that particular line and do some further processing. if ((grep(/$_/,$1)) && (grep($pattern,@myarr))) { #Do... (2 Replies)
Discussion started by: King Nothing
2 Replies

9. Shell Programming and Scripting

grep using Perl

I'm using perl to do a grep of each line in a vendor file and find its occurrences in a specific directory. Any values found is saved in @dir. .....(file opened, etc.) .... while ($line=<FILE>){ @dir = `grep $line * `; } It's the specific usage of the system grep that I'm having... (7 Replies)
Discussion started by: gavineq
7 Replies

10. Shell Programming and Scripting

Newbie to perl - Help with stats script

Hi, this is my first post so here goes..... I need help... I am trying to write a script to produce some stats based on a number of searches in a log file. Now i know how to do this using multiple variables which are really just greps, but I want a more efficent way of doing this as my poor... (1 Reply)
Discussion started by: ARwebble
1 Replies
Login or Register to Ask a Question