Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using "whitelist" from a file to remove entries

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using "whitelist" from a file to remove entries
# 1  
Old 06-23-2012
Using "whitelist" from a file to remove entries
Dear all,
what I need to do is extract an entries list from a file and remove some entries based on a white list present on other file, then output into result.txt file.

Example:

Code:
source.txt:
12345 text1 text2 text3 text4
123 text1 text2 text3 text4
678 text1 text2 text3 text4
987 text1 text2 text3 text4
456 text1 text2 text3 text4

Code:
whitelist.txt
123
987

Code:
output on result.txt file:
12345 text1 text2 text3 text4
678 text1 text2 text3 text4
456 text1 text2 text3 text4

What is the best and fast way to do that?
I can change the CR in whitelist.txt and put a "," like:
123,987
if this can simplify the code...

Many thanks!
Last edited by Lord Spectre; 06-23-2012 at 06:18 PM..
# 2  
Old 06-23-2012
Have you tried using grep?

Code:
grep -v -f whitelist.txt source.txt >result.txt

# 3  
Old 06-23-2012
Quote:
Originally Posted by agama
Have you tried using grep?
Code:
grep -v -f whitelist.txt source.txt >result.txt

Does't work:
Code:
[root@localhost ]# grep -v -f whitelist.txt source.txt
678
456

BTW, is slightly more complicated. I've update the first post, since in the source file there's some other datas....
# 4  
Old 06-23-2012
Slight modification to agama's suggestion.. Try:
Code:
grep -vxf whitelist.txt source.txt

--edit--
OK, I see the original post got changed in the mean time...

Try:
Code:
grep -vwf whitelist.txt source.txt

But that could still go wrong if a number is present in the bla bla after field 1. So this would be safer:
Code:
awk 'NR==FNR{A[$1]; next}!($1 in A)' whitelist.txt source.txt



--
On Solaris use /usr/xpg4/bin/awk rather than awk
Last edited by Scrutinizer; 06-23-2012 at 06:27 PM..
These 2 Users Gave Thanks to Scrutinizer For This Post:
agama Lord Spectre
# 5  
Old 06-23-2012
Scrutinizer, you're absolutely the best, it works perfect!! Smilie
Code:
[root@localhost ]# awk 'NR==FNR{A[$1]; next}!($1 in A)' whitelist.txt source.txt
12345 text1 text2 text3 text4
678 text1 text2 text3 text4
456 text1 text2 text3 text4

Do you think is it possible to make something like this? :
Code:
awk 'NR==FNR{A[$1]; next}!($1 in A)' whitelist.txt source.txt | while read line; do
echo $line | awk '{printf $1}'
done

What I need to do is extract values from already filtered values (eg: 1st one) line by line and create another output file like:
Code:
blabla 12345 text 
textx 678 some
texty 456 try

OK, I can output into another file the first AWK, and the loop into that file, but is there any other way using directly your AWK command ??
# 6  
Old 06-23-2012
You can do this, which will provides only the first fields and take it from there:
Code:
awk 'NR==FNR{A[$1]; next}!($1 in A){print $1}' whitelist.txt source.txt

This User Gave Thanks to Scrutinizer For This Post:
Lord Spectre
# 7  
Old 06-23-2012
Quote:
Originally Posted by Scrutinizer
You can do this, which will provides only the first fields and take it from there:
Code:
awk 'NR==FNR{A[$1]; next}!($1 in A){print $1}' whitelist.txt source.txt

It works like the previous one, obviously with only the first field.
The problem is that I cannot add more text to output, like this:
Code:
awk 'NR==FNR{A[$1]; next}!($1 in A) blabla {print $1} text' whitelist.txt source.txt

Expected result:
Code:
blabla 12345 text
blabla 678 text
blabla 456 text

Last edited by Lord Spectre; 06-23-2012 at 08:05 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

2. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

3. Shell Programming and Scripting

Cant get awk 1liner to remove duplicate lines from Delimited file, get "event not found" error..help

Hi, I am on a Solaris8 machine If someone can help me with adjusting this awk 1 liner (turning it into a real awkscript) to get by this "event not found error" ...or Present Perl solution code that works for Perl5.8 in the csh shell ...that would be great. ****************** ... (3 Replies)
Discussion started by: andy b
3 Replies

4. Shell Programming and Scripting

how to use "cut" or "awk" or "sed" to remove a string

logs: "/home/abc/public_html/index.php" "/home/abc/public_html/index.php" "/home/xyz/public_html/index.php" "/home/xyz/public_html/index.php" "/home/xyz/public_html/index.php" how to use "cut" or "awk" or "sed" to get the following result: abc abc xyz xyz xyz (8 Replies)
Discussion started by: timmywong
8 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. Shell Programming and Scripting

Remove ":" and join lines in outline file

I have a vim outliner file like this: Title title 2 :Testing now :testing 2 :testing 3 title 3 :testing :ttt :ttg Is there a way to use a script or command to remove... (7 Replies)
Discussion started by: jostber
7 Replies

7. Shell Programming and Scripting

How to remove "New line characters" and "spaces" at a time

Dear friends, following is the output of a script from which I want to remove spaces and new-line characters. Example:- Line1 abcdefghijklmnopqrstuvwxyz Line2 mnopqrstuvwxyzabcdefghijkl Line3 opqrstuvwxyzabcdefdefg Here in above example, at every starting line there is a “tab” &... (4 Replies)
Discussion started by: anushree.a
4 Replies

8. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies
Login or Register to Ask a Question