Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Fast algorithm to compare an IP address against a list of IP sections?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Fast algorithm to compare an IP address against a list of IP sections?
# 1  
Old 11-25-2010
Fast algorithm to compare an IP address against a list of IP sections?
I have two files:
Code:
file1:
41.138.128.0    41.138.159.255  location
41.138.160.0    41.138.191.255  location
41.138.192.0    41.138.207.255  location
41.138.208.0    41.138.223.255  location
41.138.224.0    41.138.239.255  location
41.138.240.0    41.138.255.255  location
41.138.32.0     41.138.63.255   location
41.138.64.0     41.138.71.255   location
41.138.72.0     41.138.79.255   location
41.138.80.0     41.138.87.255   location
.....



file2:
41.138.208.3    information
41.138.211.23    information
.....

file1, containing IP section information, has about 10,000 rows, and file2 containing an IP and other information, has 30,000 rows and is growing.
Now I want to fetch the "location" field from file1 based on the IP field from file2, and combine "location" field with "information" field from these two files, I know I can convert all this IPs to an unsinged integer, and compare the IP field from file2 against the IP section in file1, but this kind of comparison is rather inefficient, I need a FAST algorithm to do this, AWK would be favored.

Anyone has an idea?
Thank you in advance.
# 2  
Old 11-25-2010
What is the field2 from file1 used for?

A straightforward approach using awk would be to ( if second field from file1 is not being used ) create an associative array with field1 and field3, then parse through the second file and check if the entry in the second file is there in the associative array, if there print out the value from the associative array.
# 3  
Old 11-25-2010
Are IP addresses in file2 always Legal IP?

So shouldn't have IP like: 41.138.208.356
# 4  
Old 11-26-2010
Quote:
Originally Posted by matrixmadhan
What is the field2 from file1 used for?

A straightforward approach using awk would be to ( if second field from file1 is not being used ) create an associative array with field1 and field3, then parse through the second file and check if the entry in the second file is there in the associative array, if there print out the value from the associative array.
Sorry, I didn't elaborate on my problem. field1 and field2 from file1 are Legal IP addresses forming a section(e.g. from 111.111.111.0 to 111.111.111.255). I want to get the 'location' filed from file1 given an IP address(the first field from file2) falling within the section.
The IP sections in file1 are sorted.
# 5  
Old 11-26-2010
Not very efficient， still need go through file1 every time and only save time by break function, if find it.

Code:
$ cat file1
41.138.128.0    41.138.159.255  location1
41.138.160.0    41.138.191.255  location2
41.138.192.0    41.138.207.255  location3
41.138.208.0    41.138.223.255  location4
41.138.224.0    41.138.239.255  location5
41.138.240.0    41.138.255.255  location6
41.138.32.0     41.138.63.255   location7
41.138.64.0     41.138.71.255   location8
41.138.72.0     41.138.79.255   location9
41.138.80.0     41.138.87.255   location10

$ cat file2
41.138.208.3    information
41.138.80.23    information
41.138.11.23    information
11.138.11.23    information

awk '
NR==FNR{split($1,s,".");split($2,e,".");a[NR]=$3;b[NR]=s[1] FS s[2];c[NR]=s[3];d[NR]=e[3];i=NR;next}
{  split($1,ip,".")
   for (j=1;j<=i;j++) 
      if (ip[1] FS ip[2]==b[j] && ip[3]>=c[j] && ip[3]<=d[j]) { print $0 FS a[j];break}
}' file1 file2

41.138.208.3    information location4
41.138.80.23    information location10

# 6  
Old 11-26-2010
Quote:
Originally Posted by rdcwayx
Not very efficient， still need go through file1 every time and only save time by break function, if find it.

Code:
$ cat file1
41.138.128.0    41.138.159.255  location1
41.138.160.0    41.138.191.255  location2
41.138.192.0    41.138.207.255  location3
41.138.208.0    41.138.223.255  location4
41.138.224.0    41.138.239.255  location5
41.138.240.0    41.138.255.255  location6
41.138.32.0     41.138.63.255   location7
41.138.64.0     41.138.71.255   location8
41.138.72.0     41.138.79.255   location9
41.138.80.0     41.138.87.255   location10

$ cat file2
41.138.208.3    information
41.138.80.23    information
41.138.11.23    information
11.138.11.23    information

awk '
NR==FNR{split($1,s,".");split($2,e,".");a[NR]=$3;b[NR]=s[1] FS s[2];c[NR]=s[3];d[NR]=e[3];i=NR;next}
{  split($1,ip,".")
   for (j=1;j<=i;j++) 
      if (ip[1] FS ip[2]==b[j] && ip[3]>=c[j] && ip[3]<=d[j]) { print $0 FS a[j];break}
}' file1 file2

41.138.208.3    information location4
41.138.80.23    information location10

This is not efficient.
Thank you for your time anyway.
# 7  
Old 11-27-2010
Can't you keep file1 exclusively as integer?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

IP address list padding with 0

Hi All, I'm hoping one of you helpful lot could come up with a way I can do the below. I've been trying to think how is best but have struggled to come up with a way. I have a list of IP addresses in the below format 1.1.1.1 102.1.2.3 102.102.1.10 102.102.102.1 I need to compare... (4 Replies)
Discussion started by: mutley2202
4 Replies

2. Shell Programming and Scripting

List the IP address of list of servers

Hi I have a file containing server names and i can ssh to all these servers without password. Could any body suggest me how to list out IP address of all the servers? Now i am manually doing this, like ssh to each server and run "ifcong -a" command and copy the ipaddress to a excel sheet.... (5 Replies)
Discussion started by: kumar85shiv
5 Replies

3. Shell Programming and Scripting

Compare IP Address from nslookup and file

Hi, I am trying to compare IP address of a server using nslookup with IP address in a file (serverlist). And if there is a difference in IP, it should send an email. But there is some error in the pgm... Please help... #!/usr/bin/ksh for server in `cat serverlist` do old_ip=`cat... (12 Replies)
Discussion started by: vkk
12 Replies

4. UNIX for Dummies Questions & Answers

compare / difference between sub-sections of files

Hi there, I'm sure this question has been asked many times but I can't find any posts with information. How can I check the differences between say lines 20 - 200 in file1 and lines 420 - 600 in file2? Thanks in advance for any help! js (2 Replies)
Discussion started by: js8765
2 Replies

5. Shell Programming and Scripting

ksh - how to list all ip address between 2 ip address

Trying to do a ksh script that needs to list all ip address between ip address a and b .. ie. Ip address A=192.168.1.200 Ip address B=192.168.2.15 So the subnet changes from 1 to 2 but I want to list all possible ip addresses between the 2.. Which would be: 192.168.1.200... (4 Replies)
Discussion started by: frustrated1
4 Replies

6. UNIX for Advanced & Expert Users

fast way to retreive a list of lines

hi there, i have to read lines from the file, where the line is just above the pattern am looking for typically this looks like this <time-stamp>|----- <pattern am searching >...... <time-stamp>|..... <some garbage > .... the log file is big wc -l ~/log/ompe.log.20081203... (8 Replies)
Discussion started by: kiranreddy1215
8 Replies

7. Shell Programming and Scripting

script to compare two files of mac address

Hi I need to write a bash shell script. I have two separate text files. One file contains a list of MAC addresses taken from a network scan, the other contains a list of MAC addresses for our currently-managed devices. How can I compare these two files, and output a list of addresses that have... (6 Replies)
Discussion started by: borderblaster
6 Replies

8. UNIX for Dummies Questions & Answers

send email from address list and subject list

Hello, Here is my problem. there are two files. first.txt <<< contains email address ====== abc@mail.com abd@mail.com abe@mail.com second.txt <<< contains webpage links ======== http//www.test.com/abc/index.html http://www.test.com/abd/index.html http://www.test.com/abe/index.html... (2 Replies)
Discussion started by: paulds
2 Replies

9. UNIX for Dummies Questions & Answers

Slow from static IP ... fast from NAT address

Has anyone had an issue where: If I telnet into a unix machine on my network, from within my natted address, there is no speed issue. When I telnet to the same machine using the static address, it is very very slow. This SCO 5.06. There is no nameserver running. Please help. I have... (3 Replies)
Discussion started by: gseyforth
3 Replies

10. Shell Programming and Scripting

fast searching algorithm

hello, i need a searching algorithm in unix. since my input file is very bulky, so need a real fast searching algorithm, to match words. i am already using grep. (3 Replies)
Discussion started by: rochitsharma
3 Replies
Login or Register to Ask a Question