Shell Programming and Scripting

Hello! I'm using sed to perform a lots of replaces in one text file. I call it this way:

Code:

sed -f commands.txt in.txt > out.txt

commands.txt has about 1000 lines, each one is some variation of:

Code:

s/from/to/gI

And in.txt has about 300 000 lines. So the problem is that operation takes about 20 minutes to complete. Is there any way to optimize? I suppose it works not the way it should. Thanks.

Moderator's Comments:

Use code tags please, ty.

Command-wise it looks pretty optimal to me. Perhaps you could split up "in.txt" into 4 or more different parts and process them independently and in parallel in the background to 4 or more different outputfiles that could then be merged after the fact....

Thanks for your reply, I'll try it out. Though I'm still convinced it works too slow. Already tried to read from STDIN this way:

Code:

mysql -uUser -pPassword -Dbase --execute="SELECT id,str FROM in" | sed -f commands.txt > out.txt

And I still get result file in 15-20 minutes.

---------- Post updated at 10:43 PM ---------- Previous update was at 08:31 PM ----------

Well I divided in.txt into pieces each of 50K lines. It appeared to be 7 files in result (average size is 1.5-4 Mb). Then I ran several `sed` instances (separating them by &). The scheme works like charm, but still it takes about 8 minutes to parse all pieces.

I just don't get it. Total size of input text is just 20 Mb. File with sed commands is just 100 Kb. These sizes are not that big imho. But why so slow then?

Computers are rather fast. I tried both windows (2.6 GHz, 2Gb RAM) and unix (Xeon 8x2.50GHz, 1Gb RAM) servers.

I think the problem is in the I-switch. I did a couple of quick tests. On average it took 10 times longer with the I-switch than it did without. Perhaps the I switch is not necessary everywhere, or perhaps you can prune the number of combinations of uppercase and lowercase search strings, for example, instead of:

Code:

s/from/to/gI

use

Code:

s/from/to/g
s/FROM/to/g

or

Code:

s/\(from\|FROM\)/to/g

or try

Code:

sed 's/from\|FROM/to/g' file1> file2

Code:

# time sed 's/from\|FROM/to/g' file1> file2
real    0m0.269s
user    0m0.225s
sys     0m0.040s
 
# time sed 's/from/to/gi' file1> file2
real    0m0.325s
user    0m0.289s
sys     0m0.033s
 
# time sed 's/\(from\|FROM\)/to/g' file1> file2
real    0m0.791s
user    0m0.750s
sys     0m0.029s

Thanks ygemici. Confirmed, the \( \)are superfluous and take more time, so they should be left out.

Thank you Scrutinizer for your excellent ideas in forums too