Searching patterns in 1 file and deleting all lines with those patterns in 2nd file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Searching patterns in 1 file and deleting all lines with those patterns in 2nd file
# 1  
Old 08-13-2009
Searching patterns in 1 file and deleting all lines with those patterns in 2nd file

Hi Gurus,

I have a file say for ex. file1 which has 3500 lines in it which are different account numbers and another file (file2) which has 230000 lines in it. I want to read all the lines in file1 and delete all those lines from file2 which has that same pattern as in file1. I am not quite comfortable with awk, sed and perl. Please help me out. Your time is really appreciated.

Ex.
file1:
Code:
81457
742222
1089361
1098187
....

file2:
Code:
1,81457,81457,1,2,0,947,
1,742222,742222,1,2,0,2994,
1,555555555,555555555,1,2,0,1,
1,796224447,796224447,1,2,0,1,
.....

In this ex. say I want to delete lines 1 and 2 from file2 since the patterns '81457', '742222' are there in file1.

Thanks,
Toms

Last edited by vgersh99; 08-13-2009 at 09:28 AM.. Reason: code tags, PLEASE!
# 2  
Old 08-13-2009
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

---------- Post updated at 08:31 AM ---------- Previous update was at 08:28 AM ----------

Code:
nawk '
  FNR==NR {a[$0];next}
  { for(i in a) if (i ~ $0) next}
  1' file1 file2

# 3  
Old 08-13-2009
i found a perl onliner for the same , take care as it edits the original file.

Code:
TESTBOX>for i in `cat file1`
 do
 perl -ni -e "print unless /$i/" file2
 done

# 4  
Old 08-14-2009
perl:

Code:
my $reg=qr/,(81457|742222|1089361|1098187),/;
while(<DATA>){
	print if not /$reg/;
}
__DATA__
1,81457,81457,1,2,0,947,
1,742222,742222,1,2,0,2994,
1,555555555,555555555,1,2,0,1,
1,796224447,796224447,1,2,0,1,

# 5  
Old 08-14-2009
Thanks a lot... Saved a lot of time....
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash - Find files excluding file patterns and subfolder patterns

Hello. For a given folder, I want to select any files find $PATH1 -f \( -name "*" but omit any files like pattern name ! -iname "*.jpg" ! -iname "*.xsession*" ..... \) and also omit any subfolder like pattern name -type d \( -name "/etc/gconf/gconf.*" -o -name "*cache*" -o -name "*Cache*" -o... (2 Replies)
Discussion started by: jcdole
2 Replies

2. Shell Programming and Scripting

How to search 2 separate patterns from a 2nd file.?

Hi Guys, Need your urgent support please. I have a file with 3 separate strings separated by a comma and 2nd file which has a sentence where I can find these 3 strings. I need to find sentences which do not have these strings and maybe redirect it to a 3rd file. All the 3 strings will occur... (3 Replies)
Discussion started by: danish0909
3 Replies

3. Shell Programming and Scripting

Deleting the lines exist between the Mattched Patterns

Hi, I have a file with the content: for name in \ sree\ rama\ laila\ srihari\ vicky\ john do echo $name done I need to remove all the name lines that exist between for (first line) and do line so that i can replace with new names. Output file should look like: (3 Replies)
Discussion started by: raosr020
3 Replies

4. Shell Programming and Scripting

Deleting lines above the patterns

I want to delete 1 line above the paatern and 3 line below the pattern and the pattern line itself, on the whole 5 lines. If there are three patterns what to do and the final text file to be captured in a new file. (3 Replies)
Discussion started by: razen
3 Replies

5. Shell Programming and Scripting

Deleting lines above the patterns

I want to delete 1 line above the paatern and 3 lines below the pattern and the pattern line itself, on the whole 5 lines. If there are three patterns to be deleted what to do and the final text file to be captured in a new file. (1 Reply)
Discussion started by: razen
1 Replies

6. Shell Programming and Scripting

reading lines from a file between two search patterns

Hi, I am new to shell scripting and is working on a script to extract lines from a log file between two time stamps using awk command. After some research I used following command: awk '/01 Oct 2011/{p=1} /10 Oct 2011/{p=0} p' test.log >> tmp.log This works fine. But now i want to... (3 Replies)
Discussion started by: davidtd
3 Replies

7. Shell Programming and Scripting

deleting lines between patterns using sed or awk

hi, Here is excerpt from my xml file <!-- The custom module to do the authentication for LDAP --> </login-module> <login-module code="com.nlayers.seneca.security.LdapLogin" flag="sufficient"> <module-option... (1 Reply)
Discussion started by: sunrexstar
1 Replies

8. Shell Programming and Scripting

Searching for multiple patterns in a file

Hi All, I have a file in which i have to search for a pattern from the beginning of the file and if the pattern is found , then i have to perform a reverse search from that line to the beginning of the file to get the first occurrence of another pattern. sample input file hey what are you... (8 Replies)
Discussion started by: Kesavan
8 Replies

9. Shell Programming and Scripting

Script for searching a pattern in 5 files and deleting the patterns given

Hi All, I have written the below script that searches for the pattern in a file and delete them if present. please can some one have a look and suggest the changes in the script. #!bin/sh # The pattern that user want to add to the files echo "Enter the pattern of the redirect" read... (4 Replies)
Discussion started by: Shazin
4 Replies

10. Shell Programming and Scripting

Delete lines between two patterns without deleting the second pattern

I want to delete lines like this sed '/FROM_HERE/,/TO_HERE/d' but I would like to *not* delete the second match, i.e. the TO_HERE line. How can I achieve this? Thank you! (1 Reply)
Discussion started by: Ilja
1 Replies
Login or Register to Ask a Question