Shell Programming and Scripting

I have one big file of size 9GB (big_file.txt). This big file has sentences and paragraphs like any usual English document. I have another file consisting of replacement strings for sed to use. The file name is replace.sed and each entry in one line looks like this:

Code:

s/\<shout\>/shout/g
s/\<b is for\>/b_is_for/g
s/\<blue petrossa \>/blue_petrossa_/g
s/\<crocodile dundee\>/crocodile_dundee/g

There are 6 million such replacement strings each in a newline.

As one can see above that my objective is to replace those strings in the big_file.txt which have a space in between words with an underscore character.

I am running the code using this command:

Code:

cat big_file.txt | sed -f replace.sed > outfile

There are no issues with the code as it runs without any errors. I can understand that since I am doing large search and replacement, it will be both time and space demanding. But I cannot understand why the memory usage in the above command keeps on increasing with time and after sometime takes up the entire primary memory available in the computer?

The I used the split command to split the

Code:

big_file.txt

in smaller chunks each of 500MB. Running the same sed one liner on one of these smaller chunks only at one time also keeps on taking up the memory space.

I even tried with GNU parallel to speed up both on the large and the smaller file:

Code:

cat big_file.txt | parallel --pipe sed -f replace.sed > outfile

The above command chokes the entire computer resulting in disk thrashing. Any idea why the above script is taking too much of "ever-increasing" space? I am using BASH on Slackware.

Paralleling process won't help but increase memory congestion as every process will allocate own memory for the same operations.

Don't cat the big_file but have sed read the file directly - that might reduce system memory consumption for piping/buffering. And, try to split the replacefile and iterate the result of the first part_replace through the rest of the part_replaces.

Like (untested)

Code:

sed -f part_rep1 big_file | sed -f part_rep2 | sed -f part_rep3   etc.

Try this on smaller subsets of both data and script files.

Thanks. Running the script now. I will post again about the observations.