Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

[solved]awk count occurences in time window

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting [solved]awk count occurences in time window
# 1  
Old 06-09-2013
[solved]awk count occurences in time window
Input File
Time, KeyStation
Code:
00:00:00,000;KS_1
00:00:01,000;KS_1
00:00:02,000;KS_1
00:00:03,000;KS_1
00:00:04,000;KS_1
00:00:05,000;KS_1
00:00:06,000;KS_1
00:01:01,000;KS_1
00:01:02,000;KS_1
00:01:03,000;KS_1
00:01:04,000;KS_1
00:01:05,000;KS_1
00:01:06,000;KS_1
01:00:01,000;KS_1
01:00:02,000;KS_1
01:00:03,000;KS_1
01:00:04,000;KS_1
01:00:05,000;KS_1
01:00:06,000;KS_1

Code:
pi@raspberrypi ~ $ awk -F";" '\
function to_millis(date_str) {\
  sub(",", ":", date_str); split(date_str,tokens,":"); return ((tokens[1]*60 + tokens[2])*60+tokens[3])*1000 + tokens[4]\
}\
BEGIN { i=0} \
{t=to_millis($1);sent_times[i]=t; i=i+1; count=0; \
for (j in sent_times) {  \
diff=t-sent_times[j]; \
if(diff >= 5000){\
delete sent_times[j]\
} else{\
count=count+1\
}\
} print $1," : ", count}'  test

I want to count all the log statments that happen in a 5 second window. I have managed to write some code to do this if all the 'keystattion' are the same.

Output
Code:
00:00:00,000  :  1
00:00:01,000  :  2
00:00:02,000  :  3
00:00:03,000  :  4
00:00:04,000  :  5
00:00:05,000  :  5
00:00:06,000  :  5
00:01:01,000  :  1
00:01:02,000  :  2
00:01:03,000  :  3
00:01:04,000  :  4
00:01:05,000  :  5
00:01:06,000  :  5
01:00:01,000  :  1
01:00:02,000  :  2
01:00:03,000  :  3
01:00:04,000  :  4
01:00:05,000  :  5
01:00:06,000  :  5

However my log file contains lines with different keystations, and I would like to be able to have a seperate count for each keystation. I know I can split the files by keystations using grep and then use my script but this is quite laborios.

Input File with multiple keystation's

Code:
00:00:00,000;KS_1
00:00:01,000;KS_2
00:00:02,000;KS_2
00:00:03,000;KS_2
00:00:04,000;KS_1
00:00:05,000;KS_1
00:00:06,000;KS_1
00:01:01,000;KS_2
00:01:02,000;KS_2
00:01:03,000;KS_2
00:01:04,000;KS_1

Desired output

Code:
00:00:00,000;KS_1 :1
00:00:01,000;KS_2 : 1
00:00:02,000;KS_2 : 2
00:00:03,000;KS_2 : 3
00:00:04,000;KS_1 :2
00:00:05,000;KS_1 :3
00:00:06,000;KS_1 :4
00:01:01,000;KS_2 :1
00:01:02,000;KS_2 :2
00:01:03,000;KS_2 :3
00:01:04,000;KS_1 :1

Is it possible to create a 2D array of sent-times, I have tried this approach but havent been able to get my attempts to compile?

---------- Post updated at 10:55 AM ---------- Previous update was at 09:41 AM ----------

gawk seems to work:

Code:
gawk -F";" '\
function to_millis(date_str) {\
  sub(",", ":", date_str); split(date_str,tokens,":"); return ((tokens[1]*60 + tokens[2])*60+tokens[3])*1000 + tokens[4]\
}\
BEGIN { i=0} \
{t=to_millis($1);sent_times[$2][i]=t; i=i+1; count[$2]=0; \
for (j in sent_times[$2]) {  \
    diff=t-sent_times[$2][j]; \
    if(diff >= 5000){\
        delete sent_times[$2][j]\
    } else{\
        count[$2]=count[$2]+1\
    }\
} print $0," : ", count[$2] }'  test

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

[Solved] Awk: count occurrence of each character for every field

Hi, let's say an input looks like: A|C|C|D A|C|I|E A|B|I|C A|T|I|B as the title of the thread explains, I am trying to get something like: 1|A=4 2|C=2|B=1|T=1 3|I=3|C=1 4|D=1|E=1|C=1|B=1 i.e. a count of every character in each field (first column of output) independently, sorted... (4 Replies)
Discussion started by: beca123456
4 Replies

2. Shell Programming and Scripting

Count the occurences of strings

I have some text files in a folder f1 with 10 columns. The first five columns of a file are shown below. aab abb 263-455 263 455 aab abb 263-455 263 455 aab abb 263-455 263 455 bbb abb 26-455 26 455 bbb abb 26-455 26 455 bbb aka 264-266 264 266 bga bga 230-232 230 ... (10 Replies)
Discussion started by: gomez
10 Replies

3. Shell Programming and Scripting

awk count occurences

line number:status, market, keystation 1,SENT,EBS,1 : 1 2,DONE,REU,1 : 1 3,SENT,EBS,2 : 1 4,DONE,EBS,1 : 0 5,SENT,EBS,2 : 0 6,SENT,EBS,2 : 0 7,SENT,EBS,2 : 0 8,SENT,EBS,1 : 1 for each status, market combination I want to keep a tally of active orders. i.e if an order is SENT, then +1, if... (8 Replies)
Discussion started by: Calypso
8 Replies

4. Shell Programming and Scripting

Count number of occurences using awk

Hi Guys, I have 2 files like below file1 xx yy file2 b yy b2 xx c1 yy xx yy Now I want an idea which can count occurences of text from file1 and file2 so outbout would be kind of (9 Replies)
Discussion started by: prashant2507198
9 Replies

5. Solaris

[solved] lpstat count how many time printed

Hi guys! i've made a little mysql database to count how many time we print/day from a solaris server. To get to this info i use the command: lpstat -W completed|grep "date"after a few dates i've noticed that the amount of printout was everytime 500?! now the lpstat command keeps only the last... (0 Replies)
Discussion started by: beta17
0 Replies

6. UNIX for Dummies Questions & Answers

Count pattern occurences

hi, I have a text..and i need to find a pattern in the text and count to the no of times the pattern occured. i have used grep command ..but the problem is , it shows the occurrences of the pattern but doesn't count no of times the pattern occuries. (5 Replies)
Discussion started by: nvnni
5 Replies

7. Shell Programming and Scripting

Count occurences of string

Hi, Please help me in finding the number of occurences of the string. Example: Apple, green, blue, Apple, Orange, green, blue are the strings can be even in the next line. The o/p should look as: Word Count ----- ----- Apple 2 green 2 Orange 1 blue 2 Thanks (2 Replies)
Discussion started by: acc888
2 Replies

8. Shell Programming and Scripting

Count of matched pattern occurences by time in a log file

We have a log file, the format is similar to this: 08/04/2011 05:03:08 Connection Success 08/04/2011 05:13:18 Connection Success 08/04/2011 05:23:28 Connection Fail 08/04/2011 05:33:38 Connection Success 08/04/2011 06:14:18 Connection Success 08/04/2011 06:24:28 Connection Fail 08/04/2011... (6 Replies)
Discussion started by: clu
6 Replies

9. Shell Programming and Scripting

Awk to count occurences

Hi, i am in need of an awk script to accomplish the following: Input table looks like: Student1 arts Student2 science Student3 arts Student4 science Student5 science Student6 science Student7 science Student8 science Student9 science Student10 science Student11 science... (8 Replies)
Discussion started by: saint2006
8 Replies

10. Shell Programming and Scripting

Perl - Count occurences

I have enclosed the script. I am able to find the files that contain my search string but when I try to count the occurences within the file I get zero always. Any help on this. #!/usr/bin/perl my $find = $ARGV; my $replace = $ARGV; my $glob = $ARGV; @filelist = <*$glob>; # process each... (22 Replies)
Discussion started by: TimHortons
22 Replies
Login or Register to Ask a Question