Please Help. Need Help searching for multiple stings in a file and removing them.


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Please Help. Need Help searching for multiple stings in a file and removing them.
# 1  
Old 08-18-2006
Please Help. Need Help searching for multiple stings in a file and removing them.

Please help. Here is my problem. I have 9000 lines in file a and 500,000 lines in file b. For each line in file a I need to search file b and remove that line. I am currently using the grep -v command and loading the output into a new file. However, because of the size of file b this takes an extremely long time to do and I have 50 files similiar to file b. Is there a simpler way to accomplish this. Here is a code snippet of what I have so far.

cat $1 | while read LINE
do
echo $LINE

grep -v $LINE fileName > OutputFile

cp OutputFile fineName

done
# 2  
Old 08-19-2006
You can use -i option of sed (if your sed have it), i.g.
sed -ie "/$LINE/d" fileName
or you can use similar perl -pie

Note: implementation of this feature can make temp files implicitly
# 3  
Old 08-19-2006
Perhaps use "fgrep -v -f", e.g....
Code:
$ head file[12]
==> file1 <==
aaa
bbb
ccc

==> file2 <==
111
aaa
222
ccc
333
bbb
444

$ fgrep -v -f file1 file2
111
222
333
444

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to generate adler32 stings that convert into hex stings in python 2.7?

I want to generate adler32 stings that converts into hex stings in python 2.7 (1 Reply)
Discussion started by: bigvito19
1 Replies

2. Shell Programming and Scripting

Script to replace stings in multiple text files

Good Evening Folks - Happy Friday! I have a need to replace a certain string in all .csv files from "0.00" to "#Missing" in my /app/hyp_app/files directory. Does anyone have a script they use regularly that's rather quick in performance? My files are rather large so I'm looking for a... (5 Replies)
Discussion started by: SIMMS7400
5 Replies

3. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

4. Shell Programming and Scripting

Help in searching a multiple text in zip file

Hi Gurus, i have 8 zipped files and each file is having more than 100,000 records or more. issue :- i want to search the missing text from each zipped files i have stuck here, the below command works fine if i give the value 10 for the deptno. if i have more than 1 records... (6 Replies)
Discussion started by: SeenuGuddu
6 Replies

5. UNIX for Dummies Questions & Answers

Searching for Multiple texts in a file

Hello guys, I hope anyone can help me with this ... I have a file in which i have around 6000 lines of same format text like 1234567 2345678 3456789 .................... Now what I have to do is that there I have to search these numbers in another file which contains hundreds of... (1 Reply)
Discussion started by: m_usmanayub
1 Replies

6. Shell Programming and Scripting

Searching for multiple patterns in a file

Hi All, I have a file in which i have to search for a pattern from the beginning of the file and if the pattern is found , then i have to perform a reverse search from that line to the beginning of the file to get the first occurrence of another pattern. sample input file hey what are you... (8 Replies)
Discussion started by: Kesavan
8 Replies

7. Shell Programming and Scripting

Perl, searching multiple files and printing returned line to new file

I am trying to find a way to utilise the full potential of my cpu cores and memory on my windows machine. Now, I am quite familiar with grep, however, running a Unix based OS is not an option right now. Unfortunately, the 32 bit grep for windows that I am running, I cannot run multiple... (1 Reply)
Discussion started by: Moloch
1 Replies

8. Shell Programming and Scripting

searching thru or combining multiple lines in a unix file

This is the problem actually: This regex: egrep "low debug.*\".*\"" $dbDir/alarmNotification.log is looking for data between the two quotation marks: ".*\" When I hate data like this: low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC... (0 Replies)
Discussion started by: ndedhia1
0 Replies

9. UNIX for Dummies Questions & Answers

removing multiple lines of text in a file

Hi, I'm trying to remove multiple lines of text based off a series of different words and output it to a new file The document contains a ton of data but i want to delete any line that has the following mx1.rr.biz.com or ns2.ri.biz.com i tried using grep -v filename "mx1.rr.biz.com" >... (3 Replies)
Discussion started by: spartan22
3 Replies

10. Shell Programming and Scripting

Searching and Removing File Content

Hi, I am trying to search a character in a file and remove it from that file.... My file looks something like this: test1.txt ckj12300_00|123|var1|10.2 ckj00200_12|444|var2|11.2 ckj00200_14|4556|var3|33.5 c00200_00_000|4558|var4|33.5 ckj00200_14|4553|var5|33.5... (7 Replies)
Discussion started by: rkumar28
7 Replies
Login or Register to Ask a Question