Convert ip ranges to CIDR netblock


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Convert ip ranges to CIDR netblock
# 15  
Old 12-14-2018
Quote:
Originally Posted by azdps
Thank you vgersh99 for the modified script. I'm using an Intel Celeron N2930 on my OpenBSD box so the timings are definitely different. I appreciative the time you've spent looking into this script. Hopfeuly others can find this beneficial in the future.

I wish I would have posted how I downloaded and parsed the downloaded file which was redirected to a temp file before you refined the script. I'm going to post it now to show you what my downloaded file looks like prior to me processing it using the script and gawk.

ftp -V -o - http://list.iblocklist.com/?list=ydx...chiveformat=gz | gunzip -c | grep -v '#' | sed '/^$/d' | sed 's/.*://' | sed 's/-/ - /' >> temp

Also the awk version that OpenBSD is currenly using is:
awk version 20110810

The version was obtained using the command awk -V
you can get rid of all the grep-s/sed-s and pipe it directly into the modified script (without any temp file).
Good luck.
This User Gave Thanks to vgersh99 For This Post:
# 16  
Old 12-14-2018
Thank you vgersh99. I conducted benchmarks with the way I had previously downloaded the file and used my script and compared it to just downloading the file and using your script and had similar timing results. Again thank you for your time.
# 17  
Old 12-19-2018
vgersh99 while doing some tests with your script I noticed it seems to be producing wrong output. I removed the following line from your script and the output then seems to be correct.

Code:
!/^#/ && NF {$1 = sanitize($(NF-1)); $2 = sanitize($NF)

I compared the output with your original script and with these 2 lines the output appears to be wrong:

Code:
!/^#/ && NF {$1 = sanitize($(NF-1)); $2 = sanitize($NF)}

!/^#/ && NF {print range2cidr(ip2dec($(NF-1)), ip2dec($NF))}

I piped the downloaded file like so:
Code:
ftp -V -o - http://list.iblocklist.com/?list=ydx...chiveformat=gz | gunzip -c | gawk -f ip2cidr.awk > final


Last edited by azdps; 12-19-2018 at 12:18 AM..
# 18  
Old 12-19-2018
sorry, these 2 lines should read:
Code:
!/^#/ && NF {$(NF-1)= sanitize($(NF-1)); $NF = sanitize($NF)}

!/^#/ && NF {print range2cidr(ip2dec($(NF-1)), ip2dec($NF))}

Let me know how it goes...

P.S. actually, to speed things up a bit we can combine both lines into one without making awk rebuilding $0 after each "sanitize":
Code:
!/^#/ && NF {
    f1=sanitize($(NF-1))
    f2=sanitize($NF)
    print range2cidr(ip2dec(f1), ip2dec(f2))
}


Last edited by vgersh99; 12-19-2018 at 11:53 AM..
This User Gave Thanks to vgersh99 For This Post:
# 19  
Old 12-19-2018
Thanks that fixed it. Overall there is a flaw in the script somewhere though. There are approx 470 IP ranges in the downloaded file that are converted wrong. For example these aren't converted correctly:

Code:
1.62.189.215 - 1.62.189.222
4.0.25.146 - 4.0.25.148
4.0.26.14 - 4.0.29.24
24.149.30.0 - 24.149.30.4

The script converts the above IP ranges to this:
Code:
1.62.189.222/32
4.0.25.148/32
4.0.29.24/32
24.149.30.4/32

They should have been converted to:
Code:
1.62.189.215/32
1.62.189.216/30
1.62.189.220/31
1.62.189.222/32
4.0.25.146/31
4.0.25.148/32
4.0.26.14/31
4.0.26.16/28
4.0.26.32/27
4.0.26.64/26
4.0.26.128/25
4.0.27.0/24
4.0.28.0/24
4.0.29.0/28
4.0.29.16/29
4.0.29.24/32
24.149.30.0/30
24.149.30.4/32

I discovered this issue while using ipcalc which accurately converts the IP ranges. The IP ranges in question convert to several CIDR blocks instead of one. The vast majority of IP address ranges in the file convert to only one CIDR block which are correct.

Anyone interested in fixing the script?

Last edited by azdps; 12-19-2018 at 05:59 PM..
# 20  
Old 12-20-2018
Good catch - thanks for the perseverance!
how about:
Code:
function range2cidr(ipStart, ipEnd, result, bits, mask, newip) {
    bits = 1
    mask = 1
    while (bits < 32) {
        newip = or(ipStart, mask)
        if ((newip>ipEnd) || ((lshift(rshift(ipStart,bits),bits)) != ipStart)) {
           bits--
           mask = rshift(mask,1)
           break
        }
        bits++
        mask = lshift(mask,1)+1
    }
    newip = or(ipStart, mask)
    bits = 32 - bits
    result = (result)?result ORS dec2ip(ipStart) "/" bits : dec2ip(ipStart) "/" bits
    if (newip < ipEnd) result = range2cidr(newip + 1, ipEnd,result)
    return result
}

# convert dotted quads to long decimal ip
#       int ip2dec("192.168.0.15")
#
function ip2dec(ip, slice) {
        split(ip, slice, ".")
        return (slice[1] * 2^24) + (slice[2] * 2^16) + (slice[3] * 2^8) + slice[4]
}

# convert decimal long ip to dotted quads
#       str dec2ip(1171259392)
#
function dec2ip(dec, ip, quad) {
        for (i=3; i>=1; i--) {
                quad = 256^i
                ip = ip int(dec/quad) "."
                dec = dec%quad
        }
        return ip dec
}

function sanitize(ip) {
        split(ip, slice, ".")
        return slice[1]/1 "." slice[2]/1 "." slice[3]/1 "." slice[4]/1
}

BEGIN{
        #FS=" |-|:"
        # to match BOTH formats: 'ip ip' AND the one in ydxerpxkpcfqjaybcssw file
        FS="[- :]"
}

# sanitize ip's
!/^#/ && NF {
  f1= sanitize($(NF-1))
  f2= sanitize($NF)
  print range2cidr(ip2dec(f1), ip2dec(f2))
}

END {print ""}


Last edited by vgersh99; 12-20-2018 at 11:40 AM..
This User Gave Thanks to vgersh99 For This Post:
# 21  
Old 12-21-2018
I tested the new script with these test IP ranges and it didn't correctly convert the IP ranges to CIDR blocks so I initially thought it wasn't working correctly. The script does not work with this specific field separator " - ":
Code:
1.62.189.215 - 1.62.189.222
4.0.25.146 - 4.0.25.148
4.0.26.14 - 4.0.29.24
24.149.30.0 - 24.149.30.4

--- Post updated 12-21-18 at 04:23 AM ---

Edit:

I removed the space between the IP address range from the test IP ranges and the script correctly converted the the IP ranges.
Code:
1.62.189.215-1.62.189.222
4.0.25.146-4.0.25.148
4.0.26.14-4.0.29.24
24.149.30.0-24.149.30.4

I also copied the test IP ranges directly from the downloaded file in question without any changes to each line and it converts properly.

Code:
# List distributed by iblocklist.com

botnet on China Unicom Heilongjiang:1.62.189.215-1.62.189.222
s0-0.ciscoseattle.bbnplanet.net:4.0.25.146-4.0.25.148
p1-0.cisco.bbnplanet.net:4.0.26.14-4.0.29.24
CEDAR FALLS UTILITIES:24.149.30.0-24.149.30.4

I added space hyphen space to the field separators and it converts my first test IP range correctly.
Code:
BEGIN{
        FS="-| - |:"
}

Thank you! The script works absolutely great! I compared the script results against ipcalc results and they were exactly the same.

Last edited by azdps; 12-21-2018 at 10:24 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. What is on Your Mind?

Blocked A6-Index and Entire AWS Netblock

Weary of seeing our load average go up to 50+, I just did a major block on these networks (stats over a less than 20 min interval): https://www.unix.com/members/1-albums215-picture866.png (3 Replies)
Discussion started by: Neo
3 Replies

2. Shell Programming and Scripting

Convert ip ranges to CIDR netblocks

Hi, Recently I had to convert a 280K lines of ip ranges to the CIDR notation and generate a file to be used by ipset (netfilter) for ip filtering. Input file: 000.000.000.000 - 000.255.255.255 , 000 , invalid ip 001.000.064.000 - 001.000.127.255 , 000 , XXXXX 001.000.245.123 -... (10 Replies)
Discussion started by: ripat
10 Replies

3. Shell Programming and Scripting

How to change ip addressing format from CIDR notation to netmask and vice versa?

Hi all, I would appreciate if someone could share how to convert CIDR notation to netmask and vice versa. The value below is just an example. it could be different numbers/ip addresses. Initial Output, let say file1.txt Final Output, let say file2.txt (3 Replies)
Discussion started by: type8code0
3 Replies

4. Shell Programming and Scripting

How to convert multiple number ranges into sequence?

Looking for a simple way to convert ranges to a numerical sequence that would assign the original value of the range to the individual numbers that are on the range. Thank you given data 13196-13199 0 13200 4 13201 10 13202-13207 3 13208-13210 7 desired... (3 Replies)
Discussion started by: jcue25
3 Replies

5. Shell Programming and Scripting

Values between ranges

Hi, I have two files file1 chr1_22450_22500 chr2_12300_12350 chr1_34500_34550 file2 11000_13000 15000_19000 33000_44000 If the file 1 ranges fall between file2 ranges then assign the value of file2 in column 2 to file1 output: chr2_12300_12350 11000_13000 chr1_34500_34550 ... (7 Replies)
Discussion started by: Diya123
7 Replies

6. UNIX for Dummies Questions & Answers

Need help filling in ranges

I have a list of about 200,000 lines in a text file that look like this: 1 1 120 1 80 200 1 150 270 5 50 170 5 100 220 5 300 420 The first column is an identifier, the next 2 columns are a range (always 120 value range) I'm trying fill in the values of those ranges, and remove... (4 Replies)
Discussion started by: knott76
4 Replies

7. Programming

How to parse IP range in CIDR format in C

Hello everybody, I'm coding a network program and i need it to "understand" ip ranges, but i don't know how to make to parse an IP CIDR range, let's say "172.16.10.0/24" to work with the specified IP range. I've found a program which does it, but i don't understand the code. Here is the... (3 Replies)
Discussion started by: semash!
3 Replies

8. Shell Programming and Scripting

date ranges

Hi, Please anyone help to achive this using perl or unix scripting . This is date in my table 20090224,based on the date need to check the files,If file exist for that date then increment by 1 for that date and check till max date 'i.e.20090301 and push those files . files1_20090224... (2 Replies)
Discussion started by: akil
2 Replies

9. Shell Programming and Scripting

Get IP list from CIDR

Dear Srs :-) I'm looking for a shell script, that given a network in CIDR format it lists all IPs, for example: Preferredly a shell script, but a Perl, Python, C, etc.. is also welcome :-) I have been looking in sipcalc, ipcalc, etc.. options but this feature is not implemented :-( ... (10 Replies)
Discussion started by: Santi
10 Replies
Login or Register to Ask a Question