grep first occurrence but continue to next entry in patternfile


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers grep first occurrence but continue to next entry in patternfile
# 1  
Old 06-07-2011
grep first occurrence but continue to next entry in patternfile

I have 1300 files (SearchFiles0001.txt, SearchFiles0002.txt, etc.) , each with 650,000 lines, tab-delimited data.

I have a pattern file, with about 1000 lines with a single word. Each single word is found in the 1300 files once.

If I grep -f PatternFile.txt SearchFiles*.txt >OutputFile.txt
the search takes forever, because it searches 650,000 lines x 1300 files for each of the 1000 words.

If I grep -f PatternFile.txt -m 1 SearchFiles*.txt >OutputFile.txt
the search stops after the first line of PatternFile.txt, and I get an OutputFile.txt with 1300 lines: the first entry of PatternFile.txt, but none of the other 999.

How do I make the search faster (by finding the first occurrence of my search parameter, and then moving on to the next search parameter in PatternFile.txt)?
# 2  
Old 06-08-2011
Code:
for i in SearchFiles*.txt
do
 for j in `cat PatternFile.txt
 do
  grep -m 1 $j $i >> OutputFile.txt
 done
done

# 3  
Old 06-09-2011
not there yet

I'm getting nothing in my .data file.

When I set -x -v, I see that the program is cycling through each FinalReport file just fine, but nothing is getting written. So I'm guessing the problem is with the 'cat'.
(part of the set -x set -v output:
+ for i in './TryingAgain_Files/TryingAgain_FinalReport*.txt'
+ for j in ''\''cat ./SearchPattern13.txt'\'''
+ grep -m 1 cat ./SearchPattern13.txt ./TryingAgain_Files/TryingAgain_FinalReport1095.txt
+ for i in './TryingAgain_Files/TryingAgain_FinalReport*.txt'
+ for j in ''\''cat ./SearchPattern13.txt'\'''
+ grep -m 1 cat ./SearchPattern13.txt ./TryingAgain_Files/TryingAgain_FinalReport1096.txt)

I also have grepped using the SearchPattern13.txt and it does pull out the data (but I stop it because it would take all day).

for i in ./TryingAgain_Files/TryingAgain_FinalReport*.txt
do
for j in 'cat ./SearchPattern13.txt'
do
grep -m 1 $j $i >>650hgdp13.data
done
done
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep util last occurrence

I have file contents /tmp/x/abc.txt /home/bin/backup/sys/a.log I need this output: /tmp/x/ /home/bin/backup/sys/ Can somebody please help me out Please use CODE tags as required by forum rules! (3 Replies)
Discussion started by: jhonnyrip
3 Replies

2. UNIX for Dummies Questions & Answers

How to grep for first entry in a file?

Hello friends, I have a question. Sometimes I have to search for an entry in a file that is repeated thousands of times. Can you tell me how to search so that i get limited results? For example: file: myfile.txt grep "hello world" myfile.txt this above grep will generate 5000... (4 Replies)
Discussion started by: DallasT
4 Replies

3. Shell Programming and Scripting

Need to grep certain entry out of log file

This one is a bit too challenging for me... Hopefully you guys can help. Let's say I have a log file called: "$MW_HOME/user_projects/domains/IDMDomain/servers/wls_ods?/logs/wls_ods1-diagnostic.log" In this log file I want to search for "DIP-10219". When I execute this $ cat... (7 Replies)
Discussion started by: exm
7 Replies

4. UNIX for Dummies Questions & Answers

Pull out multiple lines with grep patternfile

Hi, I'm trying to get lines from a file using identifiers in the first two columns. I have used: cat MasterFile.txt | grep -f Pattern.txt and the lines I want display on screen. If I try to put them in a file the file is created but stays empty: cat MasterFile.txt | grep -f Pattern.txt... (14 Replies)
Discussion started by: FGPonce
14 Replies

5. UNIX Desktop Questions & Answers

grep a specific amount of occurrence

hey , i m trying to figure out how to do the following : i got a text file the looks like so: 1031 1031 1031 1031 1031 1031 1031 1031 16500 16500 16500 16500 1031 1031 (4 Replies)
Discussion started by: boaz733
4 Replies

6. UNIX Desktop Questions & Answers

How to grep for password file entry

How would I grep for password file entry without using grep 'username' /etc/passwd? perhaps with who? I want to create alias that will find the password file entry regardless of the user who is using it. Thanks (4 Replies)
Discussion started by: alis
4 Replies

7. UNIX for Dummies Questions & Answers

grep for password file entry

How would I grep for password file entry without using grep 'username' /etc/passwd? perhaps with who? I want to create alias that will find the password file entry regardless of the user who is using it. I am trying to get the same exact line from the file entry like: Name : Password : UserID... (7 Replies)
Discussion started by: alis
7 Replies

8. UNIX for Dummies Questions & Answers

Line number of an occurrence using grep

Hi All, is there a way to extract the line number of an occurrence using grep? I know that with the -n option it prints out the line number as well. I would like to assign the line number to a variable. Thanks, Sarah (5 Replies)
Discussion started by: f_o_555
5 Replies

9. UNIX for Dummies Questions & Answers

grep usage - exit after first occurrence

Hi, Is there any way of using grep (this may be done in awk, not sure?) that I can stop grep'n a file once I have found the first occurrence of my search string. Looking through grep man pages -q will exit without printing the lines after the first match, but I need the output. I have... (5 Replies)
Discussion started by: nhatch
5 Replies

10. Shell Programming and Scripting

Grep for the same occurrence or maybe Sed

Hi, I have a file that looks like this dasdjasdjoasjdoasjdoa SYN dakspodkapsdka asdasdasdasdasdasdasd SYN sdfsdfsdfsdfdf shfishifhsdifhsidhfif fsdfsdfsdfsdfs sdfsdfsdfsdsdfsdfsdff cercercercerce sdasdajsdoajsodasodoo FIN dasdaskdpasdda... (4 Replies)
Discussion started by: hcclnoodles
4 Replies
Login or Register to Ask a Question