performance of shell script ( grep command)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting performance of shell script ( grep command)
# 1  
Old 12-23-2009
performance of shell script ( grep command)

Hi,

I have to find out the run time for 40-45 different componets. These components writes in to a genreric log file in a single directory.

eg.

directory is LOG and the log file name format is generic_log_<process_id>_<date YY_MM_DD_HH_MM_SS>.log

i am taking the run time using the time mentioned in the log file name which is actuall the log file creation time (component start time) and component end time with the last modification time of the log file.

I have tested this and it is working fine for less no of file, but when i am executing it for a month the script is taking hell lot of time.

i am searching the following patterns to check run time for a partiular component

Code:
grep -wl "SOURCE=$COMPONENT" $(grep -wl "RUN_DATE=$SEARCH_DATE" $LOG/generic_log*

but as we have 40 components which runs on daily basis, so for 30 days we'll have around 1200 log files to search for 2 patterns, thats why the script is taking lot of time.

Can any one please advice me batter way to do this.

Last edited by zaxxon; 12-23-2009 at 05:58 AM.. Reason: use code tags please,
# 2  
Old 12-23-2009
You could try to run those commands in parallel:

Code:
grep -Fwl "RUN_DATE=$SEARCH_DATE" $LOG/generic_log* |
  xargs grep -Fwl "SOURCE=$COMPONENT"

# 3  
Old 12-23-2009
Is there a reason for only using xargs on one side of the pipe? I would have thought if there were too many arguments for the grep on the right the same would be true of the grep on the left, or is there some obscure reason I am missing? ( Wouldn't be the first time I am missing something... :-) )
# 4  
Old 12-23-2009
Quote:
Originally Posted by steadyonabix
Is there a reason for only using xargs on one side of the pipe? I would have thought if there were too many arguments for the grep on the right the same would be true of the grep on the left, or is there some obscure reason I am missing? ( Wouldn't be the first time I am missing something... :-) )
The xargs here is not used to avoid the argument list too long error. I'm using it to pass the input as filename(s), not as STDIN.
Consider the following:
Code:
zsh-4.3.10[t]% print -l 1 2 > infile
zsh-4.3.10[t]% cat infile 
1
2
zsh-4.3.10[t]% grep -Fwl 1 infile | grep -Fwl 2
zsh-4.3.10[t]% grep -Fwl 1 infile | xargs grep -Fwl 2
infile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Performance issue in shell script

Hi All, I am facing performance issue while rinning the LINUX shell script. I have file1 and file 2. File one is the source file and file 2 is lookup file. Need to replace if the pattern is matching in file1 with file2. The order of lookup file is important as if any match then exit... (8 Replies)
Discussion started by: ureddy
8 Replies

2. Shell Programming and Scripting

Shell Script to grep output from top command.

Hello, I want a script which would grep details from top command for specific processes. I m not sure of the PID of these processes but i know the names. $ top -c top - 16:41:55 up 160 days, 5:53, 2 users, load average: 9.36, 9.18, 8.98 Tasks: 288 total, 9 running, 279 sleeping, 0... (8 Replies)
Discussion started by: Siddheshk
8 Replies

3. HP-UX

Performance issue with 'grep' command for huge file size

I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is: while read line do emp_name=`echo $line` grep -e... (7 Replies)
Discussion started by: arb_1984
7 Replies

4. Shell Programming and Scripting

How to use grep command in shell script?

Hi, I am new to shell scripting. I had written a small code which accepts config file as parameter. In config file I mentioned a path for input files, INPUT_FILE_FOLDER=/project/dev/con/InputFile SEARCH_MASK_INPUT_FILE=ABC_CR_PQR*.* I want to use this path to get all file names from... (9 Replies)
Discussion started by: Poonamol
9 Replies

5. Shell Programming and Scripting

can anyone help with shell script command about searching word with grep command?

i want to search in the current directory all the files that contain one word for example "hello" i want to achieve it with the grep command but not with the grep * (2 Replies)
Discussion started by: aintour
2 Replies

6. Shell Programming and Scripting

Grep command in shell script

Hi, I have written the following shell script - Error_String="error" var1="| grep -v note1 | grep -v note2" grep -i $Error_String /users/mqm/Pwork/Err/*.out $var1 The above script gives error saying "grep: can't open | grep: can't open grep grep: can't open -v" etc In my program... (3 Replies)
Discussion started by: prasannasupp
3 Replies

7. Shell Programming and Scripting

Passing parameters to Shell script for GREP command

I am using grep to capture date from a file . Since i need to use the shell script for different dates ,is it possible to pass the date parameter to the shell script the Script is as below grep -E "08 Aug 2008|2008-08-08"* somefile.txt>test.txt The above script file greps the... (1 Reply)
Discussion started by: sud.tech
1 Replies

8. Shell Programming and Scripting

how to redirect the output of a grep command to a file inside a shell script

hi, i wat to get the output of a grep command in a file. but when i am trying out the same grep command in the unix prompt its working fine.. i am getting the output properly.. but when i am writing the same command inside my shell script , its just creating a new output file with no contents... (11 Replies)
Discussion started by: kripssmart
11 Replies

9. UNIX for Dummies Questions & Answers

how to get the output of a grep command to a file inside a shell script

hi, i wat to get the output of a grep command in a file. but when i am trying out the same grep command in the unix prompt its working fine.. i am getting the output properly.. but when i am writing the same command inside my shell script , its just creating a new output file with no contents... (1 Reply)
Discussion started by: kripssmart
1 Replies

10. UNIX for Advanced & Expert Users

Performance of a shell script

Hiii, I wrote a shell script for testing purpose. I have to test around 200thousand entries with the script.When i am doing only for 6000 entries its taking almost 1hour.If i test the whole testingdata it will take huge amount of time. I just want to know is it something dependent on the... (2 Replies)
Discussion started by: namishtiwari
2 Replies
Login or Register to Ask a Question