Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Counting entries in a file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Counting entries in a file
# 15  
Old 10-06-2011
I see. Here is the fix:

Code:
interval=2

awk -v s=$interval 'NR==1{min=$1}
                    {NoP[$1]++;UnIP[$1 FS $2]++;IP[$2];min=min>$1?$1:min;max=max>$1?max:$1}
                   END{for (i=min;i<=max;i=i+s)  
                         { b=i
                           {while (b<i+s) 
                                 {t+=NoP[b]
                                  for (j in IP) if (UnIP[b FS j]) x[j]
                                  b++
                                 }
                            }
                            print ++e,t,length(x)
                           t=0;u=0;delete x}
                       }' infile

# 16  
Old 10-06-2011
Its still not solving my purpose, but thanks anyways!
# 17  
Old 10-06-2011
Quote:
Originally Posted by sajal.bhatia
Its still not solving my purpose, but thanks anyways!
Hi Guy,

You need point out what's problem you faced. Please think it with positive way, people here are helping you by free.

From your previous description, it does the job.

Code:
1 5 4
2 7 4

Where is the problem, you need give the detail.
# 18  
Old 10-06-2011
Okay. I have a file with three columns. Time-stamp, IP Address and Bytes. Time-stamp is in epoch and there are multiple entries for same time-stamp, each representing a packet.

I need a awk/sed script which takes time (in secs) as the user input and give an output file with five columns (tab separated). Time-stamp, Number of packets in that interval (essentially a count of number of entries within that time interval), number of unique IPs in that interval, number of NEW IPs in that interval (as compared to the previous interval) and sum of bytes in that interval.

Hope I am clear this time.

Cheers,
# 19  
Old 10-06-2011
I go through the whole post and don't see any source files which have three columns, and if you need the result as 5 columns, the best way is to show the sample source file, and your expect output directly.

When you reply that others' codes don't resolve your problem with your non-write-down purpose, how can we think about you?
# 20  
Old 10-06-2011
Okay..the input file is in the following format
[DATA]# Epoch Time IP Address

899726401 112.254.1.0 20
899726401 112.254.1.0 10
899726402 154.162.38.0 30
899726402 112.254.1.0 40
899726402 165.161.7.0 60
899726403 101.226.38.0 20
899726403 101.226.38.0 10
899726403 101.226.38.0 30
899726403 112.254.1.0 40
899726403 144.12.40.0 20
899726404 144.12.40.0 30
899726404 1.14.4.0 10[/DATA]

So for the user input of 1sec the output should be like
[DATA]#Time-stamp No.of Packets No.of Unique IPs No.of New IPs Bytes

1 2 1 1 30
2 3 3 2 130
3 5 3 2 120
4 2 2 1 40[/DATA]

Hope I am clear.

Cheers,
# 21  
Old 10-06-2011
Can you explain what's the "No.of New IPs", column 4?

Code:
awk -v s=$interval 'NR==1{min=$1}
                    {NoP[$1]++;UnIP[$1 FS $2]++;IP[$2];min=min>$1?$1:min;max=max>$1?max:$1;byte[$1]+=$3}
                   END{for (i=min;i<=max;i=i+s)  
                         { b=i
                           {while (b<i+s) 
                                 {t+=NoP[b];u+=byte[b]
                                  for (j in IP) if (UnIP[b FS j]) x[j]
                                  b++
                                 }
                            }
                            print ++e,t,length(x),u
                           t=0;u=0;delete x}
                       }' infile

interval=1

1 2 1 30
2 3 3 130
3 5 3 120
4 2 2 40

interval=2

1 5 3 160
2 7 4 160
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Counting and print from file

Dear community, I have an already filtered log on my machine, something like: WARN 2016.03.10 10:59:01.136 logging.LogAlarmListener raise ALARMWARNINGRAISED Alarm NODE-NetworkAccessGroup.Client.41283 SERVICEDOWN-41283.WC severity WARNING raised: Service 41283.WC protocoltype client is down... (13 Replies)
Discussion started by: Lord Spectre
13 Replies

2. Shell Programming and Scripting

Need help of counting no of column of a file

Hi All , I got stuck on the below scenario.If anyone can help me ,that will be really helpful. I have a target hdfs file layout.I need to know the no of column in that file. Target_RECRD_layout { ABC_ID EN NOTNULLABLE, ABC_COUNTRY CHARACTER ENCODING ASCII NOTNULLABLE, ... (5 Replies)
Discussion started by: STCET22
5 Replies

3. Shell Programming and Scripting

Counting lines in a file using awk

I want to count lines of a file using AWK (only) and not in the END part like this awk 'END{print FNR}' because I want to use it. Does anyone know of a way? Thanks a lot. (7 Replies)
Discussion started by: guitarist684
7 Replies

4. UNIX for Dummies Questions & Answers

Counting feilds entries with Perl

Hi All, I have a small problem of counting the number of times a particular entry that exists in a horizontal string of elements and a vertical feild (column of entries). For example AATGGTCCTGExpected outputA=2 C=2 G=3 T=3 I have an idea to do this but I dont know how to do that if these entries... (1 Reply)
Discussion started by: pawannoel
1 Replies

5. Shell Programming and Scripting

Counting characters within a file

Ok say I wanted to count every Y in a data file. Then set Y as my delimiter so that I can separate my file by taking all the contents that occur BEFORE the first Y and store them in a variable so that I may use this content later on in my program. Then I could do the same thing with the next Y's... (5 Replies)
Discussion started by: puttster
5 Replies

6. Shell Programming and Scripting

Counting duplicate entries in a file using awk

Hi, I have a very big (with around 1 million entries) txt file with IPv4 addresses in the standard format, i.e. a.b.c.d The file looks like 10.1.1.1 10.1.1.1 10.1.1.1 10.1.2.4 10.1.2.4 12.1.5.6 . . . . and so on.... There are duplicate/multiple entries for some IP... (3 Replies)
Discussion started by: sajal.bhatia
3 Replies

7. Shell Programming and Scripting

Counting multiple entries in a file using awk

Hi, I have a big file (~960MB) having epoch time values (~50 million entries) which looks like 897393601 897393601 897393601 897393601 897393602 897393602 897393602 897393602 897393602 897393603 897393603 897393603 897393603 and so on....each time stamp has more than one... (6 Replies)
Discussion started by: sajal.bhatia
6 Replies

8. Programming

Counting the words in a file

Please find the below program. It contains the purpose of the program itself. /* Program : Write a program to count the number of words in a given text file */ /* Date : 12-June-2010 */ # include <stdio.h> # include <stdlib.h> # include <string.h> int main( int argc, char *argv ) {... (6 Replies)
Discussion started by: ramkrix
6 Replies

9. Shell Programming and Scripting

Help me in counting records from file

Hi, Please help me in counting the below records(1st field) from samplefile: Expected output: Count Descr ------------------------------------------- 7 Mean manager 14 ... (7 Replies)
Discussion started by: prashant43
7 Replies

10. Shell Programming and Scripting

Counting words in a file

I'm trying to figure out a way to count the number of words in the follwing file: cal 2002 > file1 Is there anyway to do this without using wc but instead using the cut command? (1 Reply)
Discussion started by: r0mulus
1 Replies
Login or Register to Ask a Question