Sponsored Content
Top Forums Shell Programming and Scripting egrep is very slow : How to improve performance Post 302166463 by vino on Tuesday 12th of February 2008 02:17:27 AM
Old 02-12-2008
Quote:
Originally Posted by hidnana
We have an egrep search in a while loop.

egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new

${PICKUP_DIR}/new_update is 210 MB file

In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other than egrep. And when we checked the timestamps, egrep is what slowing it down.

Is it possible to improve egrep's performance ? Or do we need to use perl or any other pattern search ?

Could you please help ?
Does the value of "key" and "PICKUP_DIR" change with each iteration ?

Look into the -f flag of grep.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

improve performance by using ls better than find

Hi , i'm searching for files over many Aix servers with rsh command using this request : find /dir1 -name '*.' -exec ls {} \; and then count them with "wc" but i would improve this search because it's too long and replace directly find with ls command but "ls *. " doesn't work. and... (3 Replies)
Discussion started by: Nicol
3 Replies

2. Shell Programming and Scripting

How to improve grep performance...

Hi All, I am using grep command to find string "abc" in one file . content of file is *********** abc = xyz def= lmn ************ i have given the below mentioned command to redirect the output to tmp file grep abc file | sort -u | awk '{print #3}' > out_file Then i am searching... (2 Replies)
Discussion started by: pooga17
2 Replies

3. UNIX for Dummies Questions & Answers

Improve Performance

hi someone tell me which ways i can improve disk I/O and system process performance.kindly refer some commands so i can do it on my test machine.thanks, Mazhar (2 Replies)
Discussion started by: mazhar99
2 Replies

4. Shell Programming and Scripting

Any way to improve performance of this script

I have a data file of 2 gig I need to do all these, but its taking hours, any where i can improve performance, thanks a lot #!/usr/bin/ksh echo TIMESTAMP="$(date +'_%y-%m-%d.%H-%M-%S')" function showHelp { cat << EOF >&2 syntax extreme.sh FILENAME Specify filename to parse EOF... (3 Replies)
Discussion started by: sirababu
3 Replies

5. Shell Programming and Scripting

Want to improve the performance of script

Hi All, I have written a script as follows which is taking lot of time in executing/searching only 3500 records taken as input from one file in log file of 12 GB Approximately. Working of script is read the csv file as an input having 2 arguments which are transaction_id,mobile_number and search... (6 Replies)
Discussion started by: poweroflinux
6 Replies

6. Programming

Help with improve the performance of grep

Input file: #content_1 12314345345 242467 #content_14 436677645 576577657 #content_100 3425546 56 #content_12 243254546 1232454 . . Reference file: content_100 (1 Reply)
Discussion started by: cpp_beginner
1 Replies

7. UNIX for Dummies Questions & Answers

How to improve the performance of this script?

Hi , i wrote a script to convert dates to the formate i want .it works fine but the conversion is tkaing lot of time . Can some one help me tweek this script #!/bin/bash file=$1 ofile=$2 cp $file $ofile mydates=$(grep -Po '+/+/+' $ofile) # gets 8/1/13 mydates=$(echo "$mydates" | sort |... (5 Replies)
Discussion started by: vikatakavi
5 Replies

8. Programming

Improve the performance of my C++ code

Hello, Attached is my very simple C++ code to remove any substrings (DNA sequence) of each other, i.e. any redundant sequence is removed to get unique sequences. Similar to sort | uniq command except there is reverse-complementary for DNA sequence. The program runs well with small dataset, but... (11 Replies)
Discussion started by: yifangt
11 Replies

9. Shell Programming and Scripting

Improve script - slow process with big files

Gents, Please can u help me to improve this script to be more faster, it works perfectly but for big files take a lot time to end the job.. I see the problem is in the step (while) and in this part the script takes a lot time.. Please if you can find a best way to do will be great. ... (13 Replies)
Discussion started by: jiam912
13 Replies

10. Solaris

Rsync quite slow (using very little cpu): how to improve its speed?

I have "inherited" a OmniOS (illumos based) server. I noticed rsync is significantly slower in respect to my reference, FreeBSD 12-CURRENT, running on exactly same hardware. Using same hardware, same command with same source and target disks, OmniOS r151026 gives: test@omniosce:~# time... (11 Replies)
Discussion started by: priyadarshan
11 Replies
egrep(1)																  egrep(1)

NAME
egrep - search a file for a pattern using full regular expressions SYNOPSIS
/usr/bin/egrep [-bchilnsv] [-e pattern_list] [-f file] [strings] [file...] /usr/xpg4/bin/egrep [-bchilnsvx] [-e pattern_list] [-f file] [strings] [file...] The egrep (expression grep) utility searches files for a pattern of characters and prints all lines that contain that pattern. egrep uses full regular expressions (expressions that have string values that use the full set of alphanumeric and special characters) to match the patterns. It uses a fast deterministic algorithm that sometimes needs exponential space. If no files are specified, egrep assumes standard input. Normally, each line found is copied to the standard output. The file name is printed before each line found if there is more than one input file. /usr/bin/egrep The /usr/bin/egrep utility accepts full regular expressions as described on the regexp(5) manual page, except for ( and ), ( and ), { and }, < and >, and , and with the addition of: 1. A full regular expression followed by + that matches one or more occurrences of the full regular expression. 2. A full regular expression followed by ? that matches 0 or 1 occurrences of the full regular expression. 3. Full regular expressions separated by | or by a NEWLINE that match strings that are matched by any of the expressions. 4. A full regular expression that can be enclosed in parentheses ()for grouping. Be careful using the characters $, *, [, ^, |, (, ), and in full regular expression, because they are also meaningful to the shell. It is safest to enclose the entire full regular expression in single quotes '... '. The order of precedence of operators is [], then *?+, then concatenation, then | and NEWLINE. /usr/xpg4/bin/egrep The /usr/xpg4/bin/egrep utility uses the regular expressions described in the EXTENDED REGULAR EXPRESSIONS section of the regex(5) manual page. The following options are supported for both /usr/bin/egrep and /usr/xpg4/bin/egrep: -b Precede each line by the block number on which it was found. This can be useful in locating block numbers by context (first block is 0). -c Print only a count of the lines that contain the pattern. -e pattern_list Search for a pattern_list (full regular expression that begins with a -). -f file Take the list of full regular expressions from file. -h Suppress printing of filenames when searching multiple files. -i Ignore upper/lower case distinction during comparisons. -l Print the names of files with matching lines once, separated by NEWLINEs. Does not repeat the names of files when the pat- tern is found more than once. -n Precede each line by its line number in the file (first line is 1). -s Work silently, that is, display nothing except error messages. This is useful for checking the error status. -v Print all lines except those that contain the pattern. /usr/xpg4/bin/egrep The following option is supported for /usr/xpg4/bin/egrep only: -x Consider only input lines that use all characters in the line to match an entire fixed string or regular expression to be matching lines. The following operands are supported: file A path name of a file to be searched for the patterns. If no file operands are specified, the standard input is used. /usr/bin/egrep pattern Specify a pattern to be used during the search for input. /usr/xpg4/bin/egrep pattern Specify one or more patterns to be used during the search for input. This operand is treated as if it were specified as -epattern_list. USAGE
See largefile(5) for the description of the behavior of egrep when encountering files greater than or equal to 2 Gbyte ( 2**31 bytes). See environ(5) for descriptions of the following environment variables that affect the execution of egrep: LC_COLLATE, LC_CTYPE, LC_MES- SAGES, and NLSPATH. The following exit values are returned: 0 If any matches are found. 1 If no matches are found. 2 For syntax errors or inaccessible files (even if matches were found). See attributes(5) for descriptions of the following attributes: /usr/bin/egrep +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |CSI |Not Enabled | +-----------------------------+-----------------------------+ /usr/xpg4/bin/egrep +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWxcu4 | +-----------------------------+-----------------------------+ |CSI |Enabled | +-----------------------------+-----------------------------+ fgrep(1), grep(1), sed(1), sh(1), attributes(5), environ(5), largefile(5), regex(5), regexp(5), XPG4(5) Ideally there should be only one grep command, but there is not a single algorithm that spans a wide enough range of space-time tradeoffs. Lines are limited only by the size of the available virtual memory. /usr/xpg4/bin/egrep The /usr/xpg4/bin/egrep utility is identical to /usr/xpg4/bin/grep -E (see grep(1)). Portable applications should use /usr/xpg4/bin/grep -E. 23 May 2005 egrep(1)
All times are GMT -4. The time now is 03:51 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy