Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Convert ip ranges to CIDR netblock

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Convert ip ranges to CIDR netblock
# 1  
Old 12-12-2018
Convert IP ranges to CIDR notation
2 scripts to convert IP ranges to CIDR notation using awk, gawk or mawk. The scripts are much faster than using ipcalc and will return the same results. The first script is reliably compatible with awk, gawk and mawk but is over 3 times as slow as the second script which is reliably compatible with gawk.

ip2cidr.awk - script 1
Code:
# Convert IP ranges to CIDR notation
# awk, gawk, mawk compatible

function range2cidr(ipStart, ipEnd, result, bits, mask, newip) {
    bits = 1
    mask = 1
    while (bits < 32) {
        newip = bit_or(ipStart, mask)
        if ((newip > ipEnd) || ((bit_lshift(bit_rshift(ipStart,bits),bits)) != ipStart)) {
            bits--
            mask = bit_rshift(mask,1)
            break
        }
        bits++
        mask = bit_lshift(mask,1)+1
    }
    newip = bit_or(ipStart, mask)
    bits = 32 - bits
    result = (result)?result ORS dec2ip(ipStart) "/" bits : dec2ip(ipStart) "/" bits
    if (newip < ipEnd) result = range2cidr(newip + 1, ipEnd,result)
    return result
}

# convert dotted quads to long decimal ip
# int ip2dec("192.168.0.15")
#
function ip2dec(ip, slice) {
    split(ip, slice, /[.]/)
    return (slice[1] * 2^24) + (slice[2] * 2^16) + (slice[3] * 2^8) + slice[4]
}

# convert decimal long ip to dotted quads
# str dec2ip(1171259392)
#
function dec2ip(dec, ip, quad) {
    for (i=3; i>=1; i--) {
        quad = 256^i
        ip = ip int(dec/quad) "."
        dec = dec%quad
    }
    return ip dec
}

# Bitwise OR of var1 and var2
function bit_or(a, b, r, i, c) {
    for (r=i=0;i<32;i++) {
        c = 2 ^ i
        if ((int(a/c) % 2) || (int(b/c) % 2)) r += c
    }
    return r
}

# Rotate bytevalue left x times
function bit_lshift(var, x) {
    while(x--) var*=2;
    return var;
}

# Rotate bytevalue right x times
function bit_rshift(var, x) {
    while(x--) var=int(var/2);
    return var;
}

function sanitize(ip) {
    split(ip, slice, /[.]/)
    return slice[1]/1 "." slice[2]/1 "." slice[3]/1 "." slice[4]/1
}

BEGIN{
    FS=" - |-|:"
}

# sanitize ip's
!/^#/ && NF {
    f1 = sanitize($(NF-1))
    f2 = sanitize($NF)
    print range2cidr(ip2dec(f1), ip2dec(f2))
}

END {print ""}

Benchmarks processing file containing approximately 236K IP ranges:
  • mawk - 1m 2s
  • gawk - 1m 49s
  • awk - 2m 35s
  • ipcalc - 15m 32s

ip2cidr.awk - script 2
Code:
# Convert IP ranges to CIDR notation
# gawk compatible

function range2cidr(ipStart, ipEnd, result, bits, mask, newip) {
    bits = 1
    mask = 1
    while (bits < 32) {
        newip = or(ipStart, mask)
        if ((newip > ipEnd) || ((lshift(rshift(ipStart,bits),bits)) != ipStart)) {
            bits--
            mask = rshift(mask,1)
            break
        }
        bits++
        mask = lshift(mask,1)+1
    }
    newip = or(ipStart, mask)
    bits = 32 - bits
    result = (result)?result ORS dec2ip(ipStart) "/" bits : dec2ip(ipStart) "/" bits
    if (newip < ipEnd) result = range2cidr(newip + 1, ipEnd,result)
    return result
}

# convert dotted quads to long decimal ip
# int ip2dec("192.168.0.15")
#
function ip2dec(ip, slice) {
    split(ip, slice, /[.]/)
    return (slice[1] * 2^24) + (slice[2] * 2^16) + (slice[3] * 2^8) + slice[4]
}

# convert decimal long ip to dotted quads
# str dec2ip(1171259392)
#
function dec2ip(dec, ip, quad) {
    for (i=3; i>=1; i--) {
        quad = 256^i
        ip = ip int(dec/quad) "."
        dec = dec%quad
    }
    return ip dec
}

function sanitize(ip) {
    split(ip, slice, /[.]/)
    return slice[1]/1 "." slice[2]/1 "." slice[3]/1 "." slice[4]/1
}

BEGIN{
    FS=" - |-|:"
}

# sanitize ip's
!/^#/ && NF {
    f1 = sanitize($(NF-1))
    f2 = sanitize($NF)
    print range2cidr(ip2dec(f1), ip2dec(f2))
}

END {print ""}

Benchmarks processing file containing approximately 236K IP ranges:
  • gawk - 34s

Original script from Convert ip ranges to CIDR netblocks post.

The original script did not accurately convert some IP ranges to proper CIDR notation. Thanks to vgersh99 for the changes to the scripts. They now properly convert IP ranges to CIDR notation.

Example use:
Code:
awk -f ip2cidr.awk ipranges.txt > cidr.txt
gawk -f ip2cidr.awk ipranges.txt > cidr.txt
mawk -f ip2cidr.awk ipranges.txt > cidr.txt

Last edited by Scott; 12-29-2018 at 02:01 PM.. Reason: Added one more code tag
# 2  
Old 12-12-2018
Did you search here first?

Convert ip ranges to CIDR netblocks
These 2 Users Gave Thanks to Corona688 For This Post:
azdps vgersh99
# 3  
Old 12-12-2018
Quote:
Originally Posted by Corona688
Did you search here first?

Convert ip ranges to CIDR netblocks
This thread is a treasure - stealing.
Thanks for sharing
Last edited by vgersh99; 12-13-2018 at 01:30 PM..
This User Gave Thanks to vgersh99 For This Post:
azdps
# 4  
Old 12-12-2018
After reviewing the suggested post I've come up with the following code:

Code:
#!/bin/sh

# Library with various ip manipulation functions
# convert ip ranges to CIDR notation

function range2cidr(ipStart, ipEnd,  bits, mask, newip) {
    bits = 1
    mask = 1
    while (bits < 32) {
        newip = or(ipStart, mask)
        if ((newip>ipEnd) || ((lshift(rshift(ipStart,bits),bits)) != ipStart)) {
           bits--
           mask = rshift(mask,1)
           break
        }
        bits++
        mask = lshift(mask,1)+1
    }
    newip = or(ipStart, mask)
    bits = 32 - bits
    result = dec2ip(ipStart) "/" bits
    if (newip < ipEnd) result = range2cidr(newip + 1, ipEnd)
    return result
}

# convert dotted quads to long decimal ip
#	int ip2dec("192.168.0.15")
#
function ip2dec(ip,   slice) {
	split(ip, slice, ".")
	return (slice[1] * 2^24) + (slice[2] * 2^16) + (slice[3] * 2^8) + slice[4]
}

# convert decimal long ip to dotted quads
#	str dec2ip(1171259392)
#
function dec2ip(dec,    ip, quad) {
	for (i=3; i>=1; i--) {
		quad = 256^i
		ip = ip int(dec/quad) "."
		dec = dec%quad
	}
	return ip dec
}

function sanitize(ip) {
	split(ip, slice, ".")
	return slice[1]/1 "." slice[2]/1 "." slice[3]/1 "." slice[4]/1
}


BEGIN{
	FS=" , | - "
}

# sanitize ip's
{$1 = sanitize($1); $2 = sanitize($2)}

# range with a single IP
$1==$2 {printf "%s\n", $1}

# ranges with multiple IP's
$1!=$2{print range2cidr(ip2dec($1), ip2dec($2))}

# footer
END {print "COMMIT\n"}

The ip address ranges are correctly converted to CIDR and printed to the console. Instead of printing to the console how do I send the information to a file instead? I don't need anything printed to the console. Also I was interested in mawk since might be faster than gawk. mawk doesn't have the lshift rshift etc. Can someone make the code more portable so I can use try mawk.

Thank you
# 5  
Old 12-12-2018
Why do you have #!/bin/sh at the beginning of awk code? It doesn't belong there. Hopefully it's doing nothing right now.

Code:
awk -f command inputfile > filename

This User Gave Thanks to Corona688 For This Post:
azdps
# 6  
Old 12-12-2018
Post #2 in the thread contained the written or() lshift() and rshift() functions if they're missing in the version of awk you're using.
This User Gave Thanks to vgersh99 For This Post:
azdps
# 7  
Old 12-12-2018
Quote:
Originally Posted by Corona688
Why do you have #!/bin/sh at the beginning of awk code? It doesn't belong there. Hopefully it's doing nothing right now.

Code:
awk -f command inputfile > filename

This works well. awk -f test.awk temp > done

Quote:
Originally Posted by vgersh99
Post #2 in the thread contained the written or() lshift() and rshift() functions if they're missing in the version of awk you're using.
Thanks I didn't notice that. Added it to my code.

So here is the code I ended up with:

Code:
#!/bin/awk -f

# Library with various ip manipulation functions
# convert ip ranges to CIDR notation

function range2cidr(ipStart, ipEnd,  bits, mask, newip) {
    bits = 1
    mask = 1
    while (bits < 32) {
        newip = bit_or(ipStart, mask)
        if ((newip>ipEnd) || ((bit_lshift(bit_rshift(ipStart,bits),bits)) != ipStart)) {
           bits--
           mask = bit_rshift(mask,1)
           break
        }
        bits++
        mask = bit_lshift(mask,1)+1
    }
    newip = bit_or(ipStart, mask)
    bits = 32 - bits
    result = dec2ip(ipStart) "/" bits
    if (newip < ipEnd) result = range2cidr(newip + 1, ipEnd)
    return result
}

# convert dotted quads to long decimal ip
#	int ip2dec("192.168.0.15")
#
function ip2dec(ip,   slice) {
	split(ip, slice, ".")
	return (slice[1] * 2^24) + (slice[2] * 2^16) + (slice[3] * 2^8) + slice[4]
}

# convert decimal long ip to dotted quads
#	str dec2ip(1171259392)
#
function dec2ip(dec,    ip, quad) {
	for (i=3; i>=1; i--) {
		quad = 256^i
		ip = ip int(dec/quad) "."
		dec = dec%quad
	}
	return ip dec
}

# Bitwise OR of var1 and var2
function bit_or(a, b, r, i, c) {
    for (r=i=0;i<32;i++) {
        c = 2 ^ i
        if ((int(a/c) % 2) || (int(b/c) % 2)) r += c
    }
    return r
}

# Rotate bytevalue left x times
function bit_lshift(var, x) {
  while(x--) var*=2;
  return var;
}

# Rotate bytevalue right x times
function bit_rshift(var, x) {
  while(x--) var=int(var/2);
  return var;
}

function sanitize(ip) {
	split(ip, slice, ".")
	return slice[1]/1 "." slice[2]/1 "." slice[3]/1 "." slice[4]/1
}


BEGIN{
	FS=" | - "
}

# sanitize ip's
{$1 = sanitize($1); $2 = sanitize($2)}

{print range2cidr(ip2dec($1), ip2dec($2))}

END {print ""}

Here are some benchmarks processing a file containing ip address ranges. The output file after running the script contained 236,315 lines.
  • ipcacl 15 min
  • mawk 59 sec
  • gawk 1 min 45 sec
  • awk 2 min 46 sec

Edit: Benchmarks were changed since they were wrong
Edit: Changed script since it wasn't appending /32 to a single IP address range
Edit: Changed field separator to be either a space or hyphen. See post #9.
Last edited by azdps; 12-13-2018 at 11:03 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. What is on Your Mind?

Blocked A6-Index and Entire AWS Netblock

Weary of seeing our load average go up to 50+, I just did a major block on these networks (stats over a less than 20 min interval): https://www.unix.com/members/1-albums215-picture866.png (3 Replies)
Discussion started by: Neo
3 Replies

2. Shell Programming and Scripting

Convert ip ranges to CIDR netblocks

Hi, Recently I had to convert a 280K lines of ip ranges to the CIDR notation and generate a file to be used by ipset (netfilter) for ip filtering. Input file: 000.000.000.000 - 000.255.255.255 , 000 , invalid ip 001.000.064.000 - 001.000.127.255 , 000 , XXXXX 001.000.245.123 -... (10 Replies)
Discussion started by: ripat
10 Replies

3. Shell Programming and Scripting

How to change ip addressing format from CIDR notation to netmask and vice versa?

Hi all, I would appreciate if someone could share how to convert CIDR notation to netmask and vice versa. The value below is just an example. it could be different numbers/ip addresses. Initial Output, let say file1.txt Final Output, let say file2.txt (3 Replies)
Discussion started by: type8code0
3 Replies

4. Shell Programming and Scripting

How to convert multiple number ranges into sequence?

Looking for a simple way to convert ranges to a numerical sequence that would assign the original value of the range to the individual numbers that are on the range. Thank you given data 13196-13199 0 13200 4 13201 10 13202-13207 3 13208-13210 7 desired... (3 Replies)
Discussion started by: jcue25
3 Replies

5. Shell Programming and Scripting

Values between ranges

Hi, I have two files file1 chr1_22450_22500 chr2_12300_12350 chr1_34500_34550 file2 11000_13000 15000_19000 33000_44000 If the file 1 ranges fall between file2 ranges then assign the value of file2 in column 2 to file1 output: chr2_12300_12350 11000_13000 chr1_34500_34550 ... (7 Replies)
Discussion started by: Diya123
7 Replies

6. UNIX for Dummies Questions & Answers

Need help filling in ranges

I have a list of about 200,000 lines in a text file that look like this: 1 1 120 1 80 200 1 150 270 5 50 170 5 100 220 5 300 420 The first column is an identifier, the next 2 columns are a range (always 120 value range) I'm trying fill in the values of those ranges, and remove... (4 Replies)
Discussion started by: knott76
4 Replies

7. Programming

How to parse IP range in CIDR format in C

Hello everybody, I'm coding a network program and i need it to "understand" ip ranges, but i don't know how to make to parse an IP CIDR range, let's say "172.16.10.0/24" to work with the specified IP range. I've found a program which does it, but i don't understand the code. Here is the... (3 Replies)
Discussion started by: semash!
3 Replies

8. Shell Programming and Scripting

date ranges

Hi, Please anyone help to achive this using perl or unix scripting . This is date in my table 20090224,based on the date need to check the files,If file exist for that date then increment by 1 for that date and check till max date 'i.e.20090301 and push those files . files1_20090224... (2 Replies)
Discussion started by: akil
2 Replies

9. Shell Programming and Scripting

Get IP list from CIDR

Dear Srs :-) I'm looking for a shell script, that given a network in CIDR format it lists all IPs, for example: Preferredly a shell script, but a Perl, Python, C, etc.. is also welcome :-) I have been looking in sipcalc, ipcalc, etc.. options but this feature is not implemented :-( ... (10 Replies)
Discussion started by: Santi
10 Replies
Login or Register to Ask a Question