Count of matched pattern occurences by time in a log file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Count of matched pattern occurences by time in a log file
# 1  
Old 04-08-2011
Count of matched pattern occurences by time in a log file

We have a log file, the format is similar to this:

08/04/2011 05:03:08 Connection Success
08/04/2011 05:13:18 Connection Success
08/04/2011 05:23:28 Connection Fail
08/04/2011 05:33:38 Connection Success
08/04/2011 06:14:18 Connection Success
08/04/2011 06:24:28 Connection Fail
08/04/2011 06:34:38 Connection Fail
08/04/2011 07:14:48 Connection Success
08/04/2011 07:25:18 Connection Success
08/04/2011 07:35:28 Connection Fail

I am hoping to write a shell script which can parse the file and have the output like:

08/04/2011 between 05:01 to 06:00 Success: 2 Fail: 1
08/04/2011 between 06:01 to 07:00 Success: 1 Fail: 2
08/04/2011 between 07:01 to 08:00 Success: 2 Fail: 0

Thanks very much for your time and help.

Last edited by clu; 04-08-2011 at 02:50 AM..
# 2  
Old 04-08-2011
can you give more details and the desired output you are expecting
If possible please also paste your current shell script
# 3  
Old 04-08-2011
Hi Harris,

I have posted my desired output as:

08/04/2011 between 05:01 to 06:00 Success: 2 Fail: 1
08/04/2011 between 06:01 to 07:00 Success: 1 Fail: 2
08/04/2011 between 07:01 to 08:00 Success: 2 Fail: 0

Thanks
# 4  
Old 04-08-2011
Code:
awk -F ":" '/Success/{s[$1]++;}/Fail/{f[$1]++}
END {for (i in s) {split(i,a," "); printf "%s Between %02d:00 to %02d:59 Success: %d Fail: %d\n", a[1],a[2],a[2],s[i],f[i]}}' infile

08/04/2011 Between 05:00 to 05:59 Success: 3 Fail: 1
08/04/2011 Between 06:00 to 06:59 Success: 1 Fail: 2
08/04/2011 Between 07:00 to 07:59 Success: 2 Fail: 1

# 5  
Old 04-08-2011
Perl

Code:
#!/usr/bin/perl

use Time::Local;

while(<DATA>) {
chomp;
@dt=unpack("A2 x1 A2 x1 A4 x1 A2 x1 A2 x1 A2",$_);
if ($dt[3] ne $cur_hr) {
        print substr($start_dt,1,10),"Between " ,substr($start_dt,11) ," and " , substr($end_dt,11)," Success ", $suc ," and Fail ", $fail,"\n" if ($suc || $fail);
        $start_dt=$dt[0]."/".$dt[1]."/".$dt[2]." ".$dt[3].":00:00";
        $mon=$dt[1] - 1;
        $yr= $dt[2] - 1900;
        $sec_start=timelocal(0,0,$dt[3],$dt[0],$mon,$yr);

        $sec_end=$sec_start + 3600;
        $end_dt=$dt[0]."/".$dt[1]."/".$dt[2]." ".$dt[3].":59:00";
        $suc=0;$fail=0;

}
        $cur_sec=timelocal($dt[5],$dt[4],$dt[3],$dt[0],$dt[1]-1,$dt[2]-1900);
        if ( $cur_sec >= $sec_start && $cur_sec <= $sec_end) {
                if ( /\bSuccess\b/ ) {++$suc;}else{++$fail;}
                }
$cur_hr=$dt[3];
}

print substr($start_dt,1,10),"Between " ,substr($start_dt,11) ," and " , substr($end_dt,11)," Success ", $suc ," and Fail ", $fail,"\n" if ($suc || $fail);
__DATA__
08/04/2011 05:03:08 Connection Success
08/04/2011 05:13:18 Connection Success
08/04/2011 05:23:28 Connection Fail
08/04/2011 05:33:38 Connection Success
08/04/2011 06:14:18 Connection Success
08/04/2011 06:24:28 Connection Fail
08/04/2011 06:34:38 Connection Fail
08/04/2011 07:14:48 Connection Success
08/04/2011 07:25:18 Connection Success
08/04/2011 07:35:28 Connection Fail

# 6  
Old 04-08-2011
Ruby(1.9+)
Code:
h=Hash.new(0)
File.open("file").each do |line|
    date,time,conn,status = line.split
    str = sprintf("%s|%s|%s", date,time[0,2],status)
    h[ str ] += 1
end
final={}
h.each do|k,v|
    s =  k.split("|")
    (final[ s[0,2].join("|") ] ||= [] ) << s[2]+":"+v.to_s
end
final.each{|x,y| a=x.split("|") ; puts "#{a[0]} #{a[1]}:00-#{a[1]}:59 : #{y}" }

Experimentation
Code:
$ ruby test.rb
08/04/2011 05:00-05:59 : ["Success:3", "Fail:1"]
08/04/2011 06:00-06:59 : ["Success:1", "Fail:2"]
08/04/2011 07:00-07:59 : ["Success:2", "Fail:1"]

# 7  
Old 04-08-2011
This is great, thanks very much for all your help.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count occurences of a character in a file by sorting results

Hello, I try to sort results of occurences in an array by using awk but I can't find the right command. that's why I'm asking your help ! :) Please see below the command that I run: awk '{ for ( i=1; i<=length; i++ ) arr++ }END{ for ( i in arr ) { print i, arr } }' dictionnary.txt ... (3 Replies)
Discussion started by: destin45
3 Replies

2. Shell Programming and Scripting

ksh : need to get the 4 th line above and 2 nd below the matched pattern in the log file

I have a log file as given below 012/01/21 10:29:02 (111111) Processing Job '23_369468343464564' 2012/01/21 10:29:02 (111111) Making Job '23_369468343464564.0'... 2012/01/21 10:29:04 (111111) Jobnumber '23_369468343464564' was successful 2012/01/21 10:29:04 ... (12 Replies)
Discussion started by: rpm120
12 Replies

3. Shell Programming and Scripting

How to use sed to search a particular pattern in a file backward after a pattern is matched.?

Hi, I have two files file1.txt and file2.txt. Please see the attachments. In file2.txt (which actually is a diff output between two versions of file1.txt.), I extract the pattern corresponding to 1172c1172. Now ,In file1.txt I have to search for this pattern 1172c1172 and if found, I have to... (9 Replies)
Discussion started by: saurabh kumar
9 Replies

4. Shell Programming and Scripting

[solved]awk count occurences in time window

Input File Time, KeyStation 00:00:00,000;KS_1 00:00:01,000;KS_1 00:00:02,000;KS_1 00:00:03,000;KS_1 00:00:04,000;KS_1 00:00:05,000;KS_1 00:00:06,000;KS_1 00:01:01,000;KS_1 00:01:02,000;KS_1 00:01:03,000;KS_1 00:01:04,000;KS_1 00:01:05,000;KS_1 00:01:06,000;KS_1 01:00:01,000;KS_1... (0 Replies)
Discussion started by: Calypso
0 Replies

5. Shell Programming and Scripting

Count of matched pattern occurences by minute and date in a log file

Anyone knows how to use AWK to achieve the following Sun Feb 12 00:41:01-00:41:59 Success:2 Fail:2 Sun Feb 12 00:42:01-00:42:59 Success:1 Fail:2 Sun Feb 12 01:20:01-01:20:59 Success:1 Fail:2 Mon Feb 13 22:41:01-22:41:59 Success:1 Fail:1 log file: Success Success Fail Fail ... (9 Replies)
Discussion started by: timmywong
9 Replies

6. UNIX for Dummies Questions & Answers

Count pattern occurences

hi, I have a text..and i need to find a pattern in the text and count to the no of times the pattern occured. i have used grep command ..but the problem is , it shows the occurrences of the pattern but doesn't count no of times the pattern occuries. (5 Replies)
Discussion started by: nvnni
5 Replies

7. Shell Programming and Scripting

How to count the number of occurences of this pattern?

Hi all, I have a pattern like this in a file: 123 4 56 789 234 5 67 789 121 3 56 789 222 4 65 789 321 6 90 100 478 8 40 789 243 7 80 789 How can I count the number of occurences of '789' (4th column) in this set...? Thanks for all your help! K (7 Replies)
Discussion started by: kripssmart
7 Replies

8. UNIX for Dummies Questions & Answers

Count of matched pattern occurance

In a file a pattern is occured many times randomly. Even it may appear more then once in the same line too. How i can get the number of times that pattern appeared in the file? let the file name is abc.txt and the pattern is "xyz". I used the following code: grep -ic "xyz" abc.txt but it is... (3 Replies)
Discussion started by: palash2k
3 Replies

9. HP-UX

count occurences of specific character in the file

For counting the occurences of specific character in the file I am issuing the command grep -o 'character' filename | wc -w It works in other shells but not in HP-UX as there is no option -o for grep. What do I do now? (9 Replies)
Discussion started by: superprogrammer
9 Replies

10. Shell Programming and Scripting

grep and count no of occurences in every line of a file

Hi Folks, I have a input file of the below format. ~~~OLKIT~OLKIT~1~~TBD~BEST PAGER & WIRELESS~4899 COMMON MARKET PLACE~~~DUBLIN~KS~43016~I~Y~DIRECT~D~~0 BPGRWRLS~~~OLKIT~OLKIT~1~~TBD~BEST PAGER & WIRELESS~4899 COMMON MARKET PLACE~~~DUBLIN~KS~43016~I~Y~DIRECT~D~~0... (12 Replies)
Discussion started by: srikanthgr1
12 Replies
Login or Register to Ask a Question