08-06-2011
Counting entries in a file
Hi,
I have a very large two column log file in the following format:
# Epoch Time IP Address
899726401 112.254.1.0
899726401 112.254.1.0
899726402 154.162.38.0
899726402 160.114.12.0
899726402 165.161.7.0
899726403 101.226.38.0
899726403 101.226.38.0
899726403 101.226.38.0
899726403 73.214.29.0
899726403 144.12.40.0
899726404 144.12.40.0
899726404 1.14.4.0
Each row represents a packet with a time stamp (epoch time) and the source IP address. As the granularity level is in "seconds", hence there are multiple entries for the same time stamp. So in 1st second there are two packets (from 1 IP), 2nd second three (from 3 IPs), 3rd second five (from 3 IPs) and so on.
I want to have a script using sed/awk (as the log files are quite big) which takes "time (seconds)" as the user input and gives the number of packets (instances of epoch time) and number of unique IP address within that specified time as the output.
So for e.g., if user gives 1 second as the input, the output file (3 columns) should be like:
#Time No of Packets No of Unique IPs
1 2 1
2 3 3
3 5 3
4 2 2
Similarly for user input as 2 second the output file should be like:
#Time No of Packets No of Unique IPs
1 5 4
2 7 4
PS: here 1 and 2 in the first column means first two seconds and next two seconds respectively.
Looking forward to the reply.
Thanks,
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I'm trying to figure out a way to count the number of words in the follwing file:
cal 2002 > file1
Is there anyway to do this without using wc but instead using the cut command? (1 Reply)
Discussion started by: r0mulus
1 Replies
2. Shell Programming and Scripting
Hi,
Please help me in counting the below records(1st field) from samplefile:
Expected output:
Count Descr
-------------------------------------------
7 Mean manager
14 ... (7 Replies)
Discussion started by: prashant43
7 Replies
3. Programming
Please find the below program. It contains the purpose of the program itself.
/* Program : Write a program to count the number of words in a given text file */
/* Date : 12-June-2010 */
# include <stdio.h>
# include <stdlib.h>
# include <string.h>
int main( int argc, char *argv )
{... (6 Replies)
Discussion started by: ramkrix
6 Replies
4. Shell Programming and Scripting
Hi,
I have a big file (~960MB) having epoch time values (~50 million entries) which looks like
897393601
897393601
897393601
897393601
897393602
897393602
897393602
897393602
897393602
897393603
897393603
897393603
897393603
and so on....each time stamp has more than one... (6 Replies)
Discussion started by: sajal.bhatia
6 Replies
5. Shell Programming and Scripting
Hi,
I have a very big (with around 1 million entries) txt file with IPv4 addresses in the standard format, i.e. a.b.c.d
The file looks like
10.1.1.1
10.1.1.1
10.1.1.1
10.1.2.4
10.1.2.4
12.1.5.6
.
.
.
.
and so on....
There are duplicate/multiple entries for some IP... (3 Replies)
Discussion started by: sajal.bhatia
3 Replies
6. Shell Programming and Scripting
Ok say I wanted to count every Y in a data file.
Then set Y as my delimiter so that I can separate my file by taking all the contents that occur BEFORE the first Y and store them in a variable so that I may use this content later on in my program. Then I could do the same thing with the next Y's... (5 Replies)
Discussion started by: puttster
5 Replies
7. UNIX for Dummies Questions & Answers
Hi All,
I have a small problem of counting the number of times a particular entry that exists in a horizontal string of elements and a vertical feild (column of entries). For example AATGGTCCTGExpected outputA=2 C=2 G=3 T=3 I have an idea to do this but I dont know how to do that if these entries... (1 Reply)
Discussion started by: pawannoel
1 Replies
8. Shell Programming and Scripting
I want to count lines of a file using AWK (only) and not in the END part like this awk 'END{print FNR}' because I want to use it.
Does anyone know of a way?
Thanks a lot. (7 Replies)
Discussion started by: guitarist684
7 Replies
9. Shell Programming and Scripting
Hi All ,
I got stuck on the below scenario.If anyone can help me ,that will be really helpful.
I have a target hdfs file layout.I need to know the no of column in that file.
Target_RECRD_layout
{
ABC_ID EN NOTNULLABLE,
ABC_COUNTRY CHARACTER ENCODING ASCII NOTNULLABLE,
... (5 Replies)
Discussion started by: STCET22
5 Replies
10. Shell Programming and Scripting
Dear community,
I have an already filtered log on my machine, something like:
WARN 2016.03.10 10:59:01.136 logging.LogAlarmListener raise ALARMWARNINGRAISED Alarm NODE-NetworkAccessGroup.Client.41283 SERVICEDOWN-41283.WC severity WARNING raised: Service 41283.WC protocoltype client is down... (13 Replies)
Discussion started by: Lord Spectre
13 Replies
LEARN ABOUT DEBIAN
pgasetintegerinitrange
PGASetIntegerInitRange(2) PGAPack PGASetIntegerInitRange(2)
NAME
PGASetIntegerInitRange - sets a flag to tell the initialization routines to set each integer-valued gene to a value chosen randomly from
the interval given by an upper and lower bound.
DESCRIPTION
No string initialization is done by this call.
INPUT PARAMETERS
ctx - context variable
min - array of lower bounds that define the interval the gene is initialized from
max - array of upper bounds that define the interval the gene is initialized from
OUTPUT PARAMETERS
none
SYNOPSIS
#include "pgapack.h"
void PGASetIntegerInitRange(ctx, min, max)
PGAContext *ctx
int *min
int *max
LOCATION
integer.c
EXAMPLE
Set the initialization routines to select a value for gene i
uniformly randomly from the interval [0,i]. Assumes all strings
are of the same length.
PGAContext *ctx;
int *low, *high, stringlen, i;
:
stringlen = PGAGetStringLength(ctx);
low = (int *) malloc(stringlen*sizeof(int));
high = (int *) malloc(stringlen*sizeof(int));
for(i=0;i<stringlen;i++) {
low[i] = 0;
high[i] = i
}
PGASetIntegerInitRange(ctx, low, high);
05/01/95 PGASetIntegerInitRange(2)