Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Pattern exclusion between two files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Pattern exclusion between two files
# 1  
Old 03-03-2014
Pattern exclusion between two files
I have a large file (file1) that has 96770154 lines and a smaller file (file2) that has 3 lines. I want to remove all of the strings from file1 that occur in file2.

file1 looks like this:

Code:
DOGDOGNODOGTESTCAT
CATHELLOBYEEBYEFAT
CATCATDOGDOGCATYESGOOD

file2 looks like this:

Code:
YES
NO
GOOD

The output should look like this:

Code:
DOGDOGNODOGTESTCAT
CATCATDOGDOGCATYESGOOD


What I have so far is this but the output is not 2 lines. It is instead 6 lines for whatever reason that I'm not understanding:

Code:
while read A B; do grep -v $A file1; done < file2 > out

Last edited by Scrutinizer; 03-03-2014 at 03:30 AM.. Reason: typos; [mod] quote tags to code tags
# 2  
Old 03-03-2014
Try the below
Code:
awk '{if(NR == FNR) {a[$0]} else {for (x in a) {if($0 ~ x) {print $0; next}}}}' file2 file1

This User Gave Thanks to SriniShoo For This Post:
verse123
# 3  
Old 03-03-2014
The while loop will call grep three times with a grep -v YES a grep -v NO and a grep -v GOOD and that output is concatenated.

Alternatively you could try without a while loop with:
Code:
grep -vf file2 file1

Note:
Code:
DOGDOGNODOGTESTCAT

will be left out since it has the string "NO" in it..
This User Gave Thanks to Scrutinizer For This Post:
verse123
# 4  
Old 03-03-2014
any way to output each of the output lines into their own separate files?
# 5  
Old 03-03-2014
You mean every line of the output is a separate file?
Code:
grep -vf file2 file1 | split -l 1

# 6  
Old 03-03-2014
all those match YES will go to output_YES, NO will go to output_NO...
but, if a string has both YES & NO, the below code will only send the line to output_YES.
Code:
awk '{if(NR == FNR) {a[$0]} else {for (x in a) {if($0 ~ x) {print $0 > "output_" x; next}}}}' file2 file1

If you want the string that contains both YES and NO to be sent to both the output file, use the below
Code:
awk '{if(NR == FNR) {a[$0]} else {for (x in a) {if($0 ~ x) {print $0 > "output_" x}}}}' file2 file1

# 7  
Old 03-03-2014
This isn't quite what I mean. In general if I have a while loop

Code:
while read A B ; do 
    something
    something else file1
done; < file2 > out

How can I output a file each time the while loop goes through 1 cycle. The length of the output can vary so I can't use
Code:
split -l 1

.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Problems with "grep -vf", and exclusion files

OK, this really has me bolluxed. I'm using Mac 10.6.8, and I'm trying to do simple analysis of my Apache logs in Unix. Let's call that text file "log". Now I want to remove from that file all lines with incomplete GETs, so I do grep -vf exclude log > out where the file "exclude" has... (13 Replies)
Discussion started by: Doug Lassiter
13 Replies

2. Shell Programming and Scripting

Mutual exclusion in getopts

Can i use mutual exclusion of particular args and force script to exit if both are specified? while getopts l:d: OPTS; do case $OPTS in l) VALUE1=$OPTARG;; d) VALUE2=$OPTARG;; *) echo "$USAGE" && exit 2;; # Here i want to exit if both -l and -d are specified ... (1 Reply)
Discussion started by: urello
1 Replies

3. Shell Programming and Scripting

Find required files by pattern in xml files and the change the pattern on Linux

Hello, I need to find all *.xml files that matched by pattern on Linux. I need to have written the file name on the screen and then change the pattern in the file just was found. For instance. I can start the script with arguments for keyword and for value, i.e script.sh keyword... (1 Reply)
Discussion started by: yart
1 Replies

4. Shell Programming and Scripting

Ftp mget exclusion question

In the script I am doing a mget for multiple files. How can I exclude just one file which I do not need to ftp ? (3 Replies)
Discussion started by: jville
3 Replies

5. Shell Programming and Scripting

Separating delimited file by pattern with exclusion list

I have a file with the contents below jan_t=jan;feb_t=feb;mar_t=mar;year=2010 jan_t=null;feb_t=feb;mar_t=mar;year=2010 jan_t=jan;feb_t=feb;mar_t=mar;year=2010 I want to extract out all the fields values ending with "_t" , however, i want to exclude feb_t and mar_t from the results In... (6 Replies)
Discussion started by: alienated
6 Replies

6. Shell Programming and Scripting

Using variables for exclusion in tar

Hello. I am writing a backup script which uses a function to create a tar archive for a specified directory. For some reason though I cannot get tar to exclude a specified path when passed as a variable. The problem does not seem to be with the command though as it works when entered in the... (2 Replies)
Discussion started by: mij
2 Replies

7. Shell Programming and Scripting

Delete old files but with exclusion with file list

Hello Can you please help and check what im missing on script below the goal is to delete the old files more than 7 days old but not the excluded file list inside excluded.dat file #!/bin/sh EXCLUDE=/path/to/exclude/exclude.dat FIND=/bin/find for xfile in '(read $EXCLUDE)' do $FIND... (9 Replies)
Discussion started by: angst_nu
9 Replies

8. UNIX for Dummies Questions & Answers

copying a pattern of files in one directory into other with new pattern names...

Hi, I have to copy a set of files abc* in /path/ to /path1/ as abc*_bkp. The list of files appear as follows in /path/: abc1 xyszd abc2 re2345 abcx .. . abcxyz I have to copy them (abc* files only) into /path1/ as: abc1_bkp abc2_bkp abcx_bkp .. . (6 Replies)
Discussion started by: new_learner
6 Replies

9. UNIX for Dummies Questions & Answers

Searching a file with exclusion in the search

Hello, I'm looking for a bit of help. Im trying to search a file for lines that contain white spaces at the end of the lines. This is what I'm using where $param is the path and file name and it redirects the output to a txt file : echo | grep -n ' $' $param >> $2 Is it possible to have... (8 Replies)
Discussion started by: gintreach
8 Replies

10. Shell Programming and Scripting

Exclusion List of file and directory

dear all i trying to list all files within a directory. I want to exclude all subdirectory and some files, with using below statement, but it not exclude the files which start with "&" and end with "SL" , is there any things wrong with the below statement ? TIA cd /myaccount/mydirectory... (6 Replies)
Discussion started by: ayang
6 Replies
Login or Register to Ask a Question