Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Filter a list of IPs

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Filter a list of IPs
# 1  
Old 02-07-2011
Filter(Delete) a list of IPs
Hello,
I have a dump of IPs (around 2 million) and i need to filter out(delete) 37 IPs from this list.
Here is a short list of IPs that i would need deleted
Code:
111.111.xxx.xxx 
123.123.xxx.xxx 
127.x.x.x
98.20.xx.xxx 
10.135.xxx.xxx 
11.105.xxx.xx 
100.100.xxx.xxx 
101.xxx.xx.xxx

would sed be the best options?
Can anyone assist?

Moderator's Comments:
Mod Comment Please use [code] and [/code] tags when posting code, data or logs etc. to preserve formatting and enhance readability, thanks.


thank you moderator, i will make sure to use [code] and [/code] next time
Last edited by senormarquez; 02-08-2011 at 12:12 AM.. Reason: code tags
# 2  
Old 02-07-2011
Try:
Code:
grep -vwf 37ips.txt wholelist.txt > newfile.txt

Last edited by zaxxon; 02-07-2011 at 12:51 PM.. Reason: added -w
This User Gave Thanks to zaxxon For This Post:
senormarquez
# 3  
Old 02-07-2011
I realize now that i should have expounded some more.

The ips come in 6 different files and i compile them into one. before i complile them, i "clean" the dump by deleting any trailing white spaces and all blank lines. i do this in a "FOR" statement
Code:
for file in `ls IP*.txt`
do
   sed -n '/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/p' $file >$file.1
   sed '/^$/d' $file.1 >$file.2
   sed 's/[ \t]*$//' $file.2 >$file.3
done

is it possible to use a sed statement to DELETE out these 37ips while in the "for" statement?

if not, i will go the grep route
Last edited by senormarquez; 02-07-2011 at 01:27 PM.. Reason: editing
# 4  
Old 02-07-2011
try this way:
Code:
awk 'NR==FNR{A[$0]=1;next}{if!((A[$0]))print}' deletelist mainfile

# 5  
Old 02-07-2011
Quote:
Originally Posted by yinyuemi
try this way:
Code:
awk 'NR==FNR{A[$0]=1;next}{if!((A[$0]))print}' deletelist mainfile


when i tried this i got an error
Code:
syntax error The source line is 1.
 The error context is
                 >>> NR==FNR{A[$0]=1;next}{if! <<< ((A[$0]))print}
 awk: The statement cannot be correctly parsed.
 The source line is 1.

any ideas.

I decided to add this syntax after the "for" statement is done, did that have something to do with it?

---------- Post updated at 03:34 PM ---------- Previous update was at 03:32 PM ----------

Quote:
Originally Posted by zaxxon
Try:
Code:
grep -vwf 37ips.txt wholelist.txt > newfile.txt

Since, i'm dealing with millions of IPs, this logic take a long time to complete.

i'm hoping to find something that works faster, if possible
# 6  
Old 02-07-2011
Try this:
Code:
awk 'NR==FNR{a[$0]; next}!($0 in a)' list dumpfile

# 7  
Old 02-07-2011
a little change, hopelly it works
Code:
awk 'NR==FNR{A[$0]=1;next}A[$0]!=1' deletelist mainfile
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Filter and delete numbers in a list

Hello, I've got a list of a single column numbers like 3000.66 3002.55 3062.23 3242.12 etc... I would like to delete all numbers higher than for example 3060.00 and lower than 2990.00 How can I do that? Thanks in advance (2 Replies)
Discussion started by: Board27
2 Replies

2. AIX

Get List of users+IPs

Hello! We have an AIX box (6.3), we are looking for a way to get list of logged in users and their IP. The issue we have is running who -ub shows hostname and we are wanting to get the ip. Any suggestions of ideas? Thanks in advance for any help! (4 Replies)
Discussion started by: entropy1980
4 Replies

3. Shell Programming and Scripting

Help with shell script - filter txt file full of ips

Hello again gentlemen. I would like to make a shell script to 'optimize' a plain text full of IPs. Let's suppose to have this text file: 1.192.63.253-1.192.63.253 1.0.234.46/32 1.1.128.0/17 1.116.0.0/14 1.177.1.157-1.177.1.157 1.23.22.19 1.192.61.0-1.192.61.99 8.6.6.6 I want to... (2 Replies)
Discussion started by: accolito
2 Replies

4. Shell Programming and Scripting

Get Ips from a list file

Hi Everyone, I typed a command: awk '{ print $1}' $LOGFILE | sort | uniq -c | sort -nr > $DEST/a.txt And I got file a.txt which show 6 1.1.1.1 3 2.2.2.2 2 3.3.3.3 1 4.4.4.4 Just now, I want to get exact ips which has first column > 5 to a file b.txt. In this situation, the... (5 Replies)
Discussion started by: testcase
5 Replies

5. Shell Programming and Scripting

Copying a files from a filter list and creating their associated parent directories

Hello all, I'm trying to copy all files within a specified directory to another location based on a find filter of mtime -1 (Solaris OS). The issue that I'm having is that in the destination directory, I want to retain the source directory structure while copying over only the files that have... (4 Replies)
Discussion started by: hunter55
4 Replies

6. Shell Programming and Scripting

Filter only gz files from list of subdirectories

Hi, I have a very big directory structure that consists of many sub-directories inside.There are around 50 ".gz" files under this dir structure. I want to copy all the gz files alone to a seperate location. Plz help me. (2 Replies)
Discussion started by: villain41
2 Replies

7. Shell Programming and Scripting

List of IPs & database lookup

I am trying to feed a list of IP's to do lookups from a database. My script works only for the first IP but all subsequent IPs output as 'unknown'. #!/usr/bin/php -q <? $ip = file('ip.txt'); foreach ($ip as $ip_num => $ip) { echo $ip; $out=sprintf("%u", ip2long($ip)); ... (1 Reply)
Discussion started by: hazno
1 Replies

8. Shell Programming and Scripting

Filter the file list greater then the size specified by user

HI, Can any tell me how to filter the list of files greater than the size specified by user. The size should be provided by user as an input. Regards shiva (6 Replies)
Discussion started by: shivu
6 Replies

9. AIX

Get the list, filter and delete the files on remote server

Hi, I need to login to a remote server. Go to a particular path. Get the lists of files on that path.There may be n number of files. I need to delete only those files from above created list which are 7 days older. I have achieved above using ftp protocol, but now the constraint has... (0 Replies)
Discussion started by: mail_amitnagpal
0 Replies
Login or Register to Ask a Question