Shell Programming and Scripting

Hi,

I have a very large two column log file in the following format:

# Epoch Time IP Address

899726401 112.254.1.0
899726401 112.254.1.0
899726402 154.162.38.0
899726402 160.114.12.0
899726402 165.161.7.0
899726403 101.226.38.0
899726403 101.226.38.0
899726403 101.226.38.0
899726403 73.214.29.0
899726403 144.12.40.0
899726404 144.12.40.0
899726404 1.14.4.0

Each row represents a packet with a time stamp (epoch time) and the source IP address. As the granularity level is in "seconds", hence there are multiple entries for the same time stamp. So in 1st second there are two packets (from 1 IP), 2nd second three (from 3 IPs), 3rd second five (from 3 IPs) and so on.

I want to have a script using sed/awk (as the log files are quite big) which takes "time (seconds)" as the user input and gives the number of packets (instances of epoch time) and number of unique IP address within that specified time as the output.

So for e.g., if user gives 1 second as the input, the output file (3 columns) should be like:

#Time No of Packets No of Unique IPs

1 2 1
2 3 3
3 5 3
4 2 2


Similarly for user input as 2 second the output file should be like:

#Time No of Packets No of Unique IPs

1 5 4
2 7 4

PS: here 1 and 2 in the first column means first two seconds and next two seconds respectively.

Looking forward to the reply.

Thanks,

Have a go with this:

Code:

#!/usr/bin/env ksh

awk -v bin_size=${1:-5} '
    function dump( )
    {
        if( NR == 1 )
            return;
        printf( "%3d %3d %3d\n", bin+1, total, length( unique ) );
        bin++;
    }

    {
        if( $1+0 >= next_bin )
        {
            dump( );
            next_bin = $1 + bin_size;

            delete unique;
            total = 0;
        }

        unique[$2]++
        total++;
    }
    END {
        if( total )
            dump( );
    }
'

exit

The size of the bin in seconds is passed on the command line as the only parameter to the script.

It should also have an "input file" as the one of the user arguments alongwith the "bin size". Its doesn't seem to work.

Can you check it again?

Thanks,

It works perfectly and it's a very good script (I tried to write one but I couldn't). Get your INPUTFILE from stdin like this:

Code:

./this_script 1 <INPUTFILE

Cheers

Sorry, I forgot to point out that the script assumes input from stdin -- I'm an old school programmer and generally write code to take input from stdin as this allows preprocessing of the input (through sed or grep) if needed without having to make a change to the script, or code the script to do extra work.

The script could also be modified slightly (the very last line) to allow the name of the input file to be supplied on the command line to read from stdin:

Code:

awk  -v bin_size=${1:-5} '
# body of awk from previous example 
' $2

Assuming the script is saved in foo.ksh, this allows it to be invoked both ways:

Code:

foo.ksh 1 <input-file
foo.ksh 1 input-file

Hi,

In place of "number of unique IPs" for that timeinterval, if I would like to calculate "number of NEW IPs (comparing the IPs in the current time interval to the IPs in the previous time interval), could someone suggest the modifications to the script?

Cheers,