Speed Up Grep Post: 302765351

Sponsored Content

Top Forums Shell Programming and Scripting Speed Up Grep Post 302765351 by Don Cragun on Saturday 2nd of February 2013 03:59:30 AM

02-02-2013

Registered User

Quote:

Originally Posted by jim mcnamara

For multiple files (grep -F is the same as fgrep, you should use one of these)
This runs five searches at a time in parallel. So it is approximately 5 times faster than a single thread of grep, on modern multi-core cpus.

Code:

cnt=0;
for i in catalina.out* 
do
   cnt=$(( $cnt + 1 ))
   grep -F 'xxxxxx' $i >> backup.txt
   [ $(( $cnt % 5  ))  -eq 0 ]  && wait
done
wait

Hi Jim,
Am I correct in assuming that you intended to have an & after backup.txt in this for loop?

Should the number of grep's to be run in parallel be tied to the number of CPU cores available to the process, or is grep I/O limited?
---------------------------
Added later: Are you sure that the output from grep is line-buffered? If not you could end up with output with parts of lines intermixed by running multiple greps writing to a single output file!

Last edited by Don Cragun; 02-02-2013 at 05:03 AM.. Reason: Additional question...

Don Cragun

View Public Profile for Don Cragun

Find all posts by Don Cragun

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Speed it up!

I wonder, are there any "tricks" to increase my server's access time in general?

2. UNIX for Dummies Questions & Answers

grep - speed of search

I grepped for a string from a directory of very large files. This took quite a long time (not a problem). When I grepped for a different string from the same files immediately after, the output was MUCH quicker. My question is, does anybody know why the second grep was so much quicker than the...

3. UNIX for Dummies Questions & Answers

Speed of mv vs. cp

Hi, Is mv (move) command quicker than cp (copy command)? I have large files and I want to know if mv actually copy the data to a new file then deletes the old or whether it just alters information the file system without physically moving data - Unfortuanately I don't have large files to test...

4. AIX

cpu speed

how do i determine the speed of a cpu on AIX 4.3.3 or 5.1?

5. Filesystems, Disks and Memory

dmidecode, RAM speed = "Current Speed: Unknown"

Hello, I have a Supermicro server with a P4SCI mother board running Debian Sarge 3.1. This is the "dmidecode" output related to RAM info: RAM speed information is incomplete.. "Current Speed: Unknown", is there anyway/soft to get the speed of installed RAM modules? thanks!! Regards :)...

6. Shell Programming and Scripting

Optimizing for a Speed-up

How would one go about optimizing this current .sh program so it works at a more minimal time. Such as is there a better way to count what I need than what I have done or better way to match patterns in the file? Thanks, #declare variables to be used. help=-1 count=0 JanCount=0 FebCount=0...

7. Shell Programming and Scripting

Speed up this script!

I have a script that processes a fair amount of data -- say, 25-50 megs per run. I'd like ideas on speeding it up. The code is actually just a preprocessor -- I'm using another language to do the heavy lifting. But as it happens, the preprocessing takes much more time than the final processing...

8. Filesystems, Disks and Memory

data from blktrace: read speed V.S. write speed

I analysed disk performance with blktrace and get some data: read: 8,3 4 2141 2.882115217 3342 Q R 195732187 + 32 8,3 4 2142 2.882116411 3342 G R 195732187 + 32 8,3 4 2144 2.882117647 3342 I R 195732187 + 32 8,3 4 2145 ...

9. Shell Programming and Scripting

How to speed up grep?

hi i'm greping the files with sepefic keyword, where the file is of too big. Assume there are 10 days log file each of more than 200mb. i've to grep all those files with a specific keywords. for example, 1. i'll grep for error message 2. after the i'll do one more grep for keyword...

10. Shell Programming and Scripting

Can I speed up my grep command?

I've got at least 30,000 XML files that I'm using the grep command to get their filename. Can I use the head command to grab just the beginning 8 lines and compare that instead of parsing the whole document? It would speed things up! or maybe grep -m?

LEARN ABOUT HPUX

zgrep

ZGREP(1)						      General Commands Manual							  ZGREP(1)

NAME

       zgrep - search possibly compressed files for a regular expression

SYNOPSIS

       zgrep [ grep_options ] [ -e ] pattern filename...

DESCRIPTION

       Zgrep  invokes  grep  on  compressed  or  gzipped  files.   These  grep	options  will  cause  zgrep  to  terminate  with  an  error  code:
       (-[drRzZ]|--di*|--exc*|--inc*|--rec*|--nu*).  All other options specified are passed directly to grep.  If no file is specified,  then  the
       standard input is decompressed if necessary and fed to grep.  Otherwise the given files are uncompressed if necessary and fed to grep.

       If the GREP environment variable is set, zgrep uses it as the grep program to be invoked.

EXIT CODE

       2 - An option that is not supported was specified.

AUTHOR

       Charles Levert (charles@comm.polymtl.ca)

SEE ALSO

       grep(1), gzexe(1), gzip(1), zdiff(1), zforce(1), zmore(1), znew(1)

																	  ZGREP(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Speed it up!

Discussion started by: pappous

2. UNIX for Dummies Questions & Answers

grep - speed of search

Discussion started by: davirime

3. UNIX for Dummies Questions & Answers

Speed of mv vs. cp

Discussion started by: GMMike

4. AIX

cpu speed

Discussion started by: csaunders