Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Shell script for search and replace by field Post 302732455 by Don Cragun on Saturday 17th of November 2012 07:17:07 PM
Old 11-17-2012
Quote:
Originally Posted by chandrath
Don,
Appreciate all your help!
Is there a way to improve performance of this script? Currently it's taking 1hr to process 200,000 source records with around 1000 records in rules file.

Thank you!
There are lots of ways to improve the performance of this script. Most of them involve reducing the tremendous flexibility that this script provides.
  1. Have different rules files for different input files. Use field numbers instead of field names in the rules files (getting rid of a level of indirection in every rule processed on every input line). Don't have rules in your rules file for fields that don't exist in your input file (every rule has to be evaluated for every line even if the rule can never match since it may apply in a later input file). Use a single character to indicate Equals and contains (less data to process and faster comparisons).
  2. Restrict the set of characters that appear in (or can be matched and replaced in rules). You could then use regular characters instead of character class expressions which might evaluate faster.
  3. Comment out or remove the debugging stuff from the script. (But only do this after you have finished changing the script forever and you are sure how it works.) Better would be to keep the debugging stuff in your source script, but strip out the debugging tests in a version you use for production runs.
  4. Get rid of the reject file and just log errors instead of logging all changes.
  5. Rewrite it in C (or some other language that is compiled rather than interpreted).
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

search and replace dynamic data in a shell script

Hi, I have a file that looks something like this: ... 0,6,256,87,0,0,0,1187443420 0,6,438,37,0,0,0,1187443380 0,2,0,0,0,10,0,1197140320 0,3,0,0,0,10,0,1197140875 0,2,0,0,0,23,0,1197140332 0,3,0,0,0,23,0,1197140437 0,2,0,0,0,17,0,1197140447 0,3,0,0,0,17,0,1197140543... (8 Replies)
Discussion started by: csejl
8 Replies

2. Shell Programming and Scripting

awk search and replace field

I am writing a c++ program that has many calls of pow(input,2). I now realize that this is slowing down the program and these all should be input * input for greater speed. There should be a simple way of doing this replacement throughout my file with awk, but I am not very familiar with awk.... (2 Replies)
Discussion started by: bluejayek
2 Replies

3. Shell Programming and Scripting

Perl - search and replace a particular field

Hi, I have a file having around 30 records. Each record has 5 fields delimited by PIPE. Few records in the file having Junk characters in the field2 and field4. I found the junk charcter and I tested it and replace the junk with space with the command below perl -i -p -e "s/\x00/ /g"... (1 Reply)
Discussion started by: ramkrix
1 Replies

4. Shell Programming and Scripting

Search duplicate field and replace one of them with new value

Dear All, I have file with 4 columns: 1 AA 0 21 2 BB 0 31 3 AA 0 21 4 CC 0 41 I would like to find the duplicate record based on column 2 and replace the 4th column of the duplicate by a new value. So, the output will be: 1 AA 0 21 2 BB 0 31 3 AA 0 -21 4 CC 0 41 Any suggestions... (3 Replies)
Discussion started by: ezhil01
3 Replies

5. Shell Programming and Scripting

Search a string,get line and replace with second field

Hi, I need to search for source path in file2 , as per file1 and if found get the next line and take the field value and put it in URL value of file1. In file1, NF is not same for all the lines. file1: <type source="/home/USER/Desktop" Dest="/home/USER/DIR1/Desktop" URL="ssh/path"/> <type... (8 Replies)
Discussion started by: greet_sed
8 Replies

6. Shell Programming and Scripting

Search and replace field?

I have 2 files A.txt and B.txt A.txt 3 fields and separate by a comma some,thing,florida any1,thing1,california some2,thing2,dallas just,fun,kansas B.txt has 8 fields and separate by a comma what,ever,florida-state,,,,,, some,one,dallas_state,,,,,, You will see 3rd fields are the... (5 Replies)
Discussion started by: sabercats
5 Replies

7. Shell Programming and Scripting

awk search and replace in a targeted field instead of $0

Hi I would like to apply this gawk command: gawk '{$0=gensub(/\y+\y/,"","g"); print}' file not to the whole $0 but just to the part of $0 that is between: (a number)"> and </mrk> Is it possible? thanks for your help. (4 Replies)
Discussion started by: louisJ
4 Replies

8. Shell Programming and Scripting

Search field in text file and replace value

Hi there, First of all this is my first post here. Thank you in advance for your help. What I am trying to do is the following. I have a text file where each field of each row is separated by a tabulator. Looks like this: ATOM 1 N HSE A 26 3.033 -10.429 -2.262 1.00 17.07 ... (8 Replies)
Discussion started by: doom4
8 Replies

9. Shell Programming and Scripting

Search for a value and replace other field in the same set

Hello friends, I have huge file with many sets where each "set" has few lines and each set always begins with "Set" in Sq brackets as shown above. # cat file1 (2 Replies)
Discussion started by: magnus29
2 Replies

10. UNIX for Dummies Questions & Answers

Search and replace the last field

Hi All, Seeking for your assistance on how to search and replace the last field/column. please see sample below: inputfile1.csv ="8923523434",="543623534"="afd23535623",="100"="200" ="8923523431",="543623536"="afd23535626",="101"="201"... (3 Replies)
Discussion started by: poginiks
3 Replies
PREG_FILTER(3)								 1							    PREG_FILTER(3)

preg_filter - Perform a regular expression search and replace

SYNOPSIS
mixed preg_filter (mixed $pattern, mixed $replacement, mixed $subject, [int $limit = -1], [int &$count]) DESCRIPTION
preg_filter(3) is identical to preg_replace(3) except it only returns the (possibly transformed) subjects where there was a match. For details about how this function works, read the preg_replace(3) documentation. RETURN VALUES
Returns an array if the $subject parameter is an array, or a string otherwise. If no matches are found or an error occurred, an empty array is returned when $subject is an array or NULL otherwise. EXAMPLES
Example #1 Example comparing preg_filter(3) with preg_replace(3) <?php $subject = array('1', 'a', '2', 'b', '3', 'A', 'B', '4'); $pattern = array('/d/', '/[a-z]/', '/[1a]/'); $replace = array('A:$0', 'B:$0', 'C:$0'); echo "preg_filter returns "; print_r(preg_filter($pattern, $replace, $subject)); echo "preg_replace returns "; print_r(preg_replace($pattern, $replace, $subject)); ?> The above example will output: preg_filter returns Array ( [0] => A:C:1 [1] => B:C:a [2] => A:2 [3] => B:b [4] => A:3 [7] => A:4 ) preg_replace returns Array ( [0] => A:C:1 [1] => B:C:a [2] => A:2 [3] => B:b [4] => A:3 [5] => A [6] => B [7] => A:4 ) SEE ALSO
PCRE Patterns, preg_quote(3), preg_replace(3), preg_replace_callback(3), preg_grep(3), preg_last_error(3). PHP Documentation Group PREG_FILTER(3)
All times are GMT -4. The time now is 05:06 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy