Howdy folks,

perhaps someone can help me with this problem. My knowledge of awk is not the best... but I've managed to a certain degree and now I'm stuck.

These are the steps and the format outputs, problem is written in red text after the STEP 2:


STEP 1
Unformated text file (100+ megs ) containing IP addresses
"unformated-unsorted-IP-list.txt"

Code:

#TEXTLINE1
#TEXTLINE2
1.1.1.1
2.2.2.2
1.1.1.2
1.1.1.3
3.3.3.3
4.4.4.4

|
|
V
script is called:

Code:

gawk -f convert2p2p.awk unformated-unsorted-IP-list.txt > formated-unsorted-ip-list.p2p

Code:

 
content of convert2p2p.awk script
#!/bin/sh
#this is a gawk script     
#this script strips first two lines from the file since they are not IP addresses, then adds CBLSPAMMER and 0 in front of IPs that are not in 000.000.000.000 format, finaly - it creates an array for each ip address in the unformated IP list file. (this is my problem, I want it to sort em, and aggregate the ranges, explained as you read onwards)
 
BEGIN {FS="\\.*"}
NR > 2 {printf "CBLSPAMMER:%03d.%03d.%03d.%03d-%03d.%03d.%03d.%03d\n" ,$1,$2,$3,$4,$1,$2,$3,$4}

|
|
V
output of "formated-unsorted-ip-list.p2p" file would look like this:

Code:

EXAMPLE:001.001.001.001-001.001.001.001
EXAMPLE:002.002.002.002-002.002.002.002
EXAMPLE:001.001.001.002-001.001.001.002
EXAMPLE:001.001.001.003-001.001.001.003
EXAMPLE:003.003.003.003-003.003.003.003
EXAMPLE:004.004.004.004-004.004.004.004

Explanation: IP addresses must be formated this way cause of sorting issues. The script generates the file size from original input file of some ~100MB to result file of around ~300MB. Yea, that's a HUGE BUNCH (9+ million ip addresses) of spammers and malware sources...

This step takes around a minute on my puter. Gawk is REALLY really fast - even when running under windows unxtools


STEP 2:
Sorting of the IP addresses

You will notice that the output of the previous step contains an array for each IP address, but that arrays are not sorted. This step does that by using cmsort or sort utility.

so, lets give example with cmsort - sorts 400MB file in under 25 mins.
script is called:

Code:

cmsort /Q /B /T=d:\temp formated-unsorted-ip-list.p2p formated-sorted-ip-list.p2p

the output file "formated-sorted-ip-list.p2p" would look like this:

Code:

EXAMPLE:001.001.001.001-001.001.001.001
EXAMPLE:001.001.001.002-001.001.001.002
EXAMPLE:001.001.001.003-001.001.001.003
EXAMPLE:002.002.002.002-002.002.002.002
EXAMPLE:003.003.003.003-003.003.003.003
EXAMPLE:004.004.004.004-004.004.004.004

Dum ta dum... and this is as far as I've managed.
The problem is in STEP 3

**** PROBLEM ****
HOW to make an gawk script that would check the adjenced IP addresses and group them into ranges, thus cutting down the number of arrays and significantly reducing the file size.

*****************

STEP 3:

Aggregating the formated, sorted IP list into arrays consisting of adjenced IP addresses.

calling the script that I need your help to create

Code:

 
gawk -f theuberscript.awk formated-sorted-ip-list.p2p > formated-sorted-AGGREGATED-ip-list.p2p

output of that script should convert the input from formated-sorted-ip-list.p2p into this:

Code:

EXAMPLE:001.001.001.001-001.001.001.003       (putting three arrays into one line - sure cuts down the filesize, right? hehe)
EXAMPLE:002.002.002.002-002.002.002.002
EXAMPLE:003.003.003.003-003.003.003.003
EXAMPLE:004.004.004.004-004.004.004.004

I don't mind if the STEP 1 and STEP 2 are placed in a single line... but somehow I think it would increase the time it will take to produce the sorted-p2p-formated output. Currently it takes around 25 minutes for 9+ milion arrays consisting of a single ip address.

I even tried doing this with my script:

Code:

 
BEGIN {FS="\\.*"}
NR > 2 {printf "CBLSPAMMER:%03d.%03d.%03d.1-%03d.%03d.%03d.254\r\n" ,$1,$2,$3,$1,$2,$3}

....but that unfortunately blocked helluvalot of IP addresses that were definitely NOT spammers or malware spreaders, so I can't use that method

In case you were wondering why I need this particular script... its cause I need it for protowall or peerguardian ... I am sure some of you use that software for torrents (legal ofc ), but I need it to block spammers... I hate the number of DNS queries that go from my server when checking if the inbound mail sender's ip address is in their dnsbls list.
And for curiosity sake =) wonder what happens if I load it with more then a handful of IPs to block

So, what do you say folks, can someone help me with this script? =)
Hope its not as complicated as I've presented it har har!

Best regards,

Matt