Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Grep patterns and group counts

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Grep patterns and group counts
# 1  
Old 09-03-2015
Grep patterns and group counts
Hi,

I have a continuous log file which has the following format:-

Code:
02/Sep/2015: IP 11.151.108.166 error occurred etc
03/Sep/2015: IP 11.151.108.188 error occurred etc
03/Sep/2015: IP 11.152.178.250 error occurred etc
03/Sep/2015: IP 11.188.108.176 error occurred etc
03/Sep/2015: IP 11.14.108.146 error occurred etc
03/Sep/2015: IP 11.188.178.1 error occurred etc
03/Sep/2015: IP 11.151.188.142 error occurred etc
03/Sep/2015: IP 11.151.21.188 error occurred etc

I want to create a daily report of total counts of IPs grouped by the first two sections of the IP eg 11.151 etc. So, for the above it would show:-

Code:
IP total counts for 03/Sep/2015:-

11.151 = 3 (entries)
11.152 = 1
11.188 = 2
11.14 = 1
etc

I think i'm okay doing the count by the date but i'm not sure how to grep the pattern of the IP.
I do know which IP ranges are in the log but would rather not hard code each individual pattern (e.g. grep 11.151*, grep 11.152* etc) in case new ones appear.

Thanks
Last edited by Scrutinizer; 09-03-2015 at 07:51 AM.. Reason: CODE tags
# 2  
Old 09-03-2015
Hello finn,

Following may help you in same, if order doesn't matter for you.
Code:
1st:
awk '{split($3, A,".");B[A[1]"."A[2]]++} END{for(i in B){print i OFS B[i]}}' OFS=" = "  Input_file
OR for specific date like you mentioned in you post
2nd:
awk '{if($1 ~ /^03\/Sep\/2015:$/){split($3, A,".");B[A[1]"."A[2]]++}} END{for(i in B){print i OFS B[i]}}' OFS=" = "  Input_file

Output will be as follows for without date in 1st command above.
Code:
11.14 = 1
11.188 = 2
11.151 = 4
11.152 = 1

Output will be as follows with date mentioned in 2nd command above.
Code:
11.14 = 1
11.188 = 2
11.151 = 3
11.152 = 1

Thanks,
R. Singh
Last edited by RavinderSingh13; 09-03-2015 at 07:37 AM.. Reason: Added a date condition in code now.
This User Gave Thanks to RavinderSingh13 For This Post:
finn
# 3  
Old 09-03-2015
Code:
[akshay@localhost tmp]$ cat file
02/Sep/2015: IP 11.151.108.166 error occurred etc
03/Sep/2015: IP 11.151.108.188 error occurred etc
03/Sep/2015: IP 11.152.178.250 error occurred etc
03/Sep/2015: IP 11.188.108.176 error occurred etc
03/Sep/2015: IP 11.14.108.146 error occurred etc
03/Sep/2015: IP 11.188.178.1 error occurred etc
03/Sep/2015: IP 11.151.188.142 error occurred etc
03/Sep/2015: IP 11.151.21.188 error occurred etc

Code:
[akshay@localhost tmp]$ awk -F'[ .]' '$1==dt":"{A[$3"."$4]++}END{for(i in A)print i" = "A[i]}' dt="03/Sep/2015" file
11.14 = 1
11.188 = 2
11.151 = 3
11.152 = 1

This User Gave Thanks to Akshay Hegde For This Post:
finn
# 4  
Old 09-03-2015
Perhaps a bit of Perl?

Code:
#!/usr/bin/perl
# finn.report.pl

# these two statements help detecting bugs
use strict;
use warnings;

my %dates; # to keep a record by dates

# read the file given at the command line
while(<>) {
    # extract date and ip, disregard rest 
    my ($day, undef, $nbits, undef) = split; 
    # extract ip bits wanted
    my ($netbits) = $nbits =~ /^(\d+\.\d+)/;
    # increase count of ip bits
    $dates{$day}{$netbits}++;
}

# report back to stdout
for my $d (sort keys %dates){
    print "IP total counts for $d\n";
    for my $ipp (sort keys %{$dates{$d}}){
        printf "%-8s = %s\n", $ipp, $dates{$d}{$ipp};
    }
    print "\n";
}

Run as
Code:
$ perl finn.report.pl finn.file 
IP total counts for 02/Sep/2015:
11.151   = 1

IP total counts for 03/Sep/2015:
11.14    = 1
11.151   = 3
11.152   = 1
11.188   = 2

Last edited by Aia; 09-04-2015 at 03:18 AM.. Reason: Added sort to the ip portions
This User Gave Thanks to Aia For This Post:
finn
# 5  
Old 09-04-2015
Thanks all - some great solutions there that work. That AWK is a really powerful tool.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to use grep with multiple patterns?

I am trying to grep a variable with multiple lines with multiple patterns below is the pattern list in a variable called "grouplst", each pattern is speerated by "|" grouplst="example1|example2|example3|example4|example5|example6|example7" I need to use the patterns above to grep a... (2 Replies)
Discussion started by: ajetangay
2 Replies

2. Shell Programming and Scripting

Grep patterns

Hi Experts, I have a log file like this.I need to filter the Index name and elapsed time(only created ). 06:36:39 SQL> create index XYZ_F75 on XYZ 06:36:39 2 ("GRP_ID", "_ID") parallel 64 nologging 06:36:39 3 tablespace XARGS_IDX 06:36:39 4 ; Index created. Elapsed:... (2 Replies)
Discussion started by: navsan420
2 Replies

3. Shell Programming and Scripting

need a one liner to grep a group info from /etc/group and use that result to search passwd file

/etc/group tiadm::345:mk789,po312,jo343,ju454,ko453,yx879,iy345,hn453 bin::2:root,daemon sys::3:root,bin,adm adm::4:root,daemon uucp::5:root /etc/passwd mk789:x:234:1::/export/home/dummy:/bin/sh po312:x:234:1::/export/home/dummy:/bin/sh ju454:x:234:1::/export/home/dummy:/bin/sh... (6 Replies)
Discussion started by: chidori
6 Replies

4. UNIX for Dummies Questions & Answers

Grep - various patterns

I have a file with the following text: grep \$ grep \\$ grep \\\$ grep '\$' grep '\'$' grep \\ grep \\\\ grep "\$" grep '"$' grep "$" When I perform these same commands on this file, the result are never what I would expect them to be. Could someone please comment on the results and... (3 Replies)
Discussion started by: uran101
3 Replies

5. Shell Programming and Scripting

How to group matched patterns in different files

Hi, I have a master file that i need to split into multiple files based on matched patterns. sample of my data as follows:- scaff_1 a e 123 130 c_scaff_100 scaff_1 a e 132 138 c_scaff_101 scaff_1 a e 140 150 ... (2 Replies)
Discussion started by: redse171
2 Replies

6. Shell Programming and Scripting

grep value between two patterns

Hi All, I've been trying solve this with a simple command but not having much luck. I have a file like this: Line 1: random_description 123/alert/high random_description2 356/alert/slow Line 2: random_description3 654/alert/medium Line 3: random_description4 234/alert/critical I'm... (7 Replies)
Discussion started by: joe19
7 Replies

7. Shell Programming and Scripting

Grep between 2 patterns in a line

Hi I have a file containing 1000+ lines of netlist data. I need to search for text between two key words on each line for e.g my input file has "ABC.ABC__312.deftr_0.X143.Xntys_0.\Xetabc__ABC_test_tz .X1023" "ABC.ABC__312.asjartwtsj"... (16 Replies)
Discussion started by: naveen@
16 Replies

8. Shell Programming and Scripting

grep or sed patterns

i want to match a pattern * ^Subject:.*<\hello\> for my emails to me that has 'hello' anywhere in the subject. (3 Replies)
Discussion started by: oxoxo
3 Replies

9. Shell Programming and Scripting

to grep and print their counts

suppose u have a file ACFCFACCACARCSHFARCVJVASTVAJFTVAJVGHBAJ another file A C F R then output shud be A= 9 C=7 F=3 R=2 Thanks (12 Replies)
Discussion started by: cdfd123
12 Replies

10. UNIX for Advanced & Expert Users

Group by columns and get the counts

Hi Gurus, I have a file 1|usa|hh 2|usa|ll 3|usa|vg 4|usa|vg 5|usa|vg 6|usa|vg 7|usa|ll 8|uk|nn 9|uk|bb 10|uk|bb 11|kuwait|mm 12|kuwait|jkj 13|kuwait|mm 14|dubai|hh I want to group by last two columns and get the last two recs and count. (3 Replies)
Discussion started by: sumeet
3 Replies
Login or Register to Ask a Question