Sponsored Content
Top Forums Shell Programming and Scripting Speeding up search and replace in a for loop Post 302657755 by Scrutinizer on Monday 18th of June 2012 10:27:29 AM
Old 06-18-2012
Quote:
Originally Posted by hergp
Wow, I did not expect ex to be so much more efficient than sed.

Do you have the total run-time in seconds for the three approaches too, pbluescript?
I don't think the difference lies so much in the sed/ex statements, but rather in the while loop that contains command substitutions with external cut commands, which is very costly in a shell loop.

We could replace the loop with one that does not use external commands:

Code:
while IFS=, read OLD NEW
do
  echo "s#\"$OLD\"#\"$NEW\"#g"
done < conversion.csv >commands.txt

I ran some tests on a file with 55000 entries and results were:
  • loop with cut commands: 4 minutes 55 seconds
  • loop without cut commands: 1.7 seconds

Last edited by Scrutinizer; 06-18-2012 at 11:36 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl: Search for string on line then search and replace text

Hi All, I have a file that I need to be able to find a pattern match on a line, search that line for a text pattern, and replace that text. An example of 4 lines in my file is: 1. MatchText_randomNumberOfText moreData ReplaceMe moreData 2. MatchText_randomNumberOfText moreData moreData... (4 Replies)
Discussion started by: Crypto
4 Replies

2. UNIX for Dummies Questions & Answers

Speeding up a Shell Script (find, grep and a for loop)

Hi all, I'm having some trouble with a shell script that I have put together to search our web pages for links to PDFs. The first thing I did was: ls -R | grep .pdf > /tmp/dave_pdfs.outWhich generates a list of all of the PDFs on the server. For the sake of arguement, say it looks like... (8 Replies)
Discussion started by: Dave Stockdale
8 Replies

3. Shell Programming and Scripting

awk - replace number of string length from search and replace for a serialized array

Hello, I really would appreciate some help with a bash script for some string manipulation on an SQL dump: I'd like to be able to rename "sites/WHATEVER/files" to "sites/SOMETHINGELSE/files" within the sql dump. This is quite easy with sed: sed -e... (1 Reply)
Discussion started by: otrotipo
1 Replies

4. Programming

PERL, search and replace inside foreach loop

Hello All, Im a Hardware engineer, I have written this script to automate my job. I got stuck in the following location. CODE: .. .. ... foreach $key(keys %arr_hash) { my ($loc,$ind,$add) = split /,/, $arr_hash{$key}; &create_verilog($key, $loc, $ind ,$add); } sub create_verilog{... (2 Replies)
Discussion started by: riyasnr007
2 Replies

5. UNIX for Dummies Questions & Answers

Speeding/Optimizing GREP search on CSV files

Hi all, I have problem with searching hundreds of CSV files, the problem is that search is lasting too long (over 5min). Csv files are "," delimited, and have 30 fields each line, but I always grep same 4 fields - so is there a way to grep just those 4 fields to speed-up search. Example:... (11 Replies)
Discussion started by: Whit3H0rse
11 Replies

6. Shell Programming and Scripting

perl search and replace - search in first line and replance in 2nd line

Dear All, i want to search particular string and want to replance next line value. following is the test file. search string is tmp,??? ,10:1 "???" may contain any 3 character it should remain the same and next line replace with ,10:50 tmp,123 --- if match tmp,??? then... (3 Replies)
Discussion started by: arvindng
3 Replies

7. Shell Programming and Scripting

search replace with loop and variable

Hi, could anyone help me with this, tried several times but still not getting it right or having enough grounding to do it outside of javascript: Using awk or sed or bash: need to go through a text file using a for next loop, replacing substrings in the file that consist of a potentially multi... (3 Replies)
Discussion started by: wind
3 Replies

8. Shell Programming and Scripting

Nested search in a file and replace the inner search

Hi Team, I am new to unix, please help me in this. I have a file named properties. The content of the file is : ##Mobile props east.url=https://qa.east.corp.com/prop/end west.url=https://qa.west.corp.com/prop/end south.url=https://qa.south.corp.com/prop/end... (2 Replies)
Discussion started by: tolearn
2 Replies

9. Shell Programming and Scripting

Speeding up substitutions

Hi all, I have a lookup table from which I am looking up values (from col1) and replacing them by corresponding values (from col2) in another file. lookup file a,b c,d So just replace a by b, and replace c by d. mainfile a,fvvgeggsegg,dvs a,fgeggefddddddddddg... (7 Replies)
Discussion started by: senhia83
7 Replies

10. Shell Programming and Scripting

Help speeding up script

This is my first experience writing unix script. I've created the following script. It does what I want it to do, but I need it to be a lot faster. Is there any way to speed it up? cat 'Tax_Provision_Sample.dat' | sort | while read p; do fn=`echo $p|cut -d~ -f2,4,3,8,9`; echo $p >> "$fn.txt";... (20 Replies)
Discussion started by: JohnN6
20 Replies
while(n)						       Tcl Built-In Commands							  while(n)

__________________________________________________________________________________________________________________________________________________

NAME
while - Execute script repeatedly as long as a condition is met SYNOPSIS
while test body _________________________________________________________________ DESCRIPTION
The while command evaluates test as an expression (in the same way that expr evaluates its argument). The value of the expression must a proper boolean value; if it is a true value then body is executed by passing it to the Tcl interpreter. Once body has been executed then test is evaluated again, and the process repeats until eventually test evaluates to a false boolean value. Continue commands may be exe- cuted inside body to terminate the current iteration of the loop, and break commands may be executed inside body to cause immediate termi- nation of the while command. The while command always returns an empty string. Note: test should almost always be enclosed in braces. If not, variable substitutions will be made before the while command starts execut- ing, which means that variable changes made by the loop body will not be considered in the expression. This is likely to result in an infinite loop. If test is enclosed in braces, variable substitutions are delayed until the expression is evaluated (before each loop iter- ation), so changes in the variables will be visible. For an example, try the following script with and without the braces around $x<10: set x 0 while {$x<10} { puts "x is $x" incr x } SEE ALSO
break(n), continue(n), for(n), foreach(n) KEYWORDS
boolean value, loop, test, while Tcl while(n)
All times are GMT -4. The time now is 03:12 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy