Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

How to grep faster ?

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers How to grep faster ?
# 1  
Old 06-19-2007
How to grep faster ?
Hi I have to grep for 2000 strings in a file one after the other.Say the file name is Snxx.out which has these strings.

I have to search for all the strings in the file Snxx.out one after the other.
What is the fastest way to do it ??

Note:The current grep process is taking lot of time per line as the Snxx.out is a large one.


Thanks and Regards,
Gideon.
Last edited by preethgideon; 06-19-2007 at 11:16 AM.. Reason: Additon.
# 2  
Old 06-19-2007
Gideon,
One way of doing this is to create a file with all the 2000 strings, then:
Code:
egrep -f <2000_strings_file> <input_file>

# 3  
Old 06-19-2007
Thank you very much.It works !!
# 4  
Old 06-20-2007
I have a new file which has 30k lines.
And when I used the above command it gives the error as

grep -f 30k_strings_file Sndxx.out
grep : regular expression too big

I suspect this becuase of the file size. What is the max number of lines grep can take at once ??

Also when I try egrep it works.What is the diff b/w grep and egrep ?

Thanks and Regards,
Gideon.
Last edited by preethgideon; 06-20-2007 at 01:12 PM.. Reason: correction
# 5  
Old 06-20-2007
Quote:
Originally Posted by preethgideon
I have a new file which has 30k lines.
And when I used the above command it gives the error as

grep -f 30k_strings_file Sndxx.out
grep : regular expression too big

I suspect this becuase of the file size. What is the max number of lines grep can take at once ??

Also when I try egrep it works.What is the diff b/w grep and egrep ?

Thanks and Regards,
Gideon.
I'm guessing when you grep using the 30k_strings_file, the grep is taking all strings as one search term. grep finds only what you specify, or in the case of "grep something file | grep somethingelse" it acts as an AND (find this and that..) egrep is an OR.. egrep "something|somethingelse" file (find this OR that..)
# 6  
Old 06-20-2007
Quote:
How to grep faster ??
Well you can't. Use perl instead.

Quote:
Difference between grep and egrep ??
Well egrep is nothing but an extended grep where additional regular expression metacharacters have been added.
kamitsin
# 7  
Old 06-20-2007
The "-f" option only applies to "egrep".
 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Need help for faster file read and grep in big files

I have a very big input file <inputFile1.txt> which has list of mobile no inputFile1.txt 3434343 3434323 0970978 85233 ... around 1 million records i have another file as inputFile2.txt which has some log detail big file inputFile2.txt afjhjdhfkjdhfkd df h8983 3434343 | 3483 | myout1 |... (3 Replies)
Discussion started by: reldb
3 Replies

2. UNIX for Dummies Questions & Answers

What is the faster way to grep from huge file?

Hi All, I am new to this forum and this is my first post. My requirement is like to optimize the time taken to grep the file with 40000 lines. There are two files FILEA(40000 lines) FILEB(40000 lines). The requirement is like this, both the file will be in the format below... (11 Replies)
Discussion started by: mad man
11 Replies

3. Shell Programming and Scripting

Grep -v -f and sort|diff which way is faster

Hi Gurus, I have two big files. I need to compare the different. currently, I am using sort file1 > file1_temp; sort file2 > file2_tmp diff file1_tmp file2_tmp I can use command grep -v -f file1 file2 just wondering which way is fast to compare two big files. Thanks... (4 Replies)
Discussion started by: ken6503
4 Replies

4. UNIX for Dummies Questions & Answers

Which system is faster?

i'm trying to decide if to move operations from one of these hosts to the other. but i cant decide which one of them is the most powerful. each host has 8 cpus. HOSTA processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU ... (6 Replies)
Discussion started by: SkySmart
6 Replies

5. UNIX for Dummies Questions & Answers

Why is RAID0 faster?

I have read anecdotes about people installing RAID0 (RAID - Wikipedia, the free encyclopedia) on some of their machines because it gives a performance boost. Because bandwidth on the motherboard is limited, can someone explain exactly why it should be faster? (7 Replies)
Discussion started by: figaro
7 Replies

6. UNIX for Dummies Questions & Answers

Which command will be faster? y?

i)wc -c/etc/passwd|awk'{print $1}' ii)ls -al/etc/passwd|awk'{print $5}' (4 Replies)
Discussion started by: karthi_g
4 Replies

7. Shell Programming and Scripting

Faster then cp ?

Hi , I need to copy every day about 35GB of files from one file system to another. Im using the cp command and its toke me about 25 min. I also tried to use dd command but its toke much more. Is there better option ? Regards. (6 Replies)
Discussion started by: yoavbe
6 Replies

8. UNIX for Advanced & Expert Users

faster way to loop?

Sample Log file IP.address Date&TimeStamp GET/POST URL ETC 123.45.67.89 MMDDYYYYHHMM GET myURL http://ABC.com 123.45.67.90 MMDDYYYYHHMM GET myURL http://XYZ.com I have a very huge web server log file (about 1.3GB) that contains entries like the one above. I need to get the last entries of... (9 Replies)
Discussion started by: tads98
9 Replies

9. IP Networking

Mandrake should be faster.

For some reason 8.1 Mandrake Linux seems much slower than Windows 2000 with my cable modem. DSL reports test says they conferable speed with Windows2 though. This is consistant slow with both of my boxes, at the same time. Linux used to be faster, but not with Mandrake. Any way to fix this? (17 Replies)
Discussion started by: lancest
17 Replies
Login or Register to Ask a Question