UNIX for Dummies Questions & Answers

We need to scramble data in a number of ASCII files. Some of these files are extremely large (1.2 GB). By scrambling, I mean that we need to substitute certain strings, which number around 400, with scrambled strings. An example has been given below

If "London" occurs in the file, then it needs to be substituted by "X1"

If "Frankfurt" occurs in the file, then it needs to be substituted by "X2".

We have written a Korn shell script, but there are huge performance problems as we need to check for 400 different strings. What is the best way of doing this ?.

The machine is HP-UX B.11.00 E 9000/800.

The solution suggested by Perderabo works...................
...............like LIGHTNING.

Thanks a lot for the help.

The exact best approach would depend on the details of your particular system. It always amazes me when folks ask questions without revealing what version of unix, what computer, etc. Well, I'll this a shot anyway.

The fastest way to do anything is to write a carefully designed assembly language program that will fully exploit the features available on your system. Following close behind would be writing the program in C.

As far as scripts go, the fastest way to to perform the two tranformations that you mentioned is this:

Code:

#! /usr/bin/sed -f
s/London/X1/g
s/Frankfurt/X2/g

You might call it "scramble" and run it like this:
./scramble < inputfile > outputfile

But you want to do 400 substitutions. sed will have some limit on the number of commands that it can handle. It is not likely that you can get all 400 in one script. You can probably get 100, but the exact limit depends on your version of unix. You could have 4 of these, like this:
./scramble1 < input | ./scramble2 | ./scramble3 | ./scramble4 > output
If your computer has at least 4 cpu's this might still be unbeatable by any other scripted solution.

The latest version of ksh, ksh93, has much of sed built-in. A carefully written ksh93 script that relies only on built-ins could probably beat the pipeline of sed scripts. But most folks only have ksh88 available.

Try the sed solution and see where that leaves you.

Using Sun OS 5.6..and for me the limit for sedfile usage is 199. Not 200 but 199 substitutions. I had a similar exercise once replacing a ceratin field with it's encrypted value - but I had around 10,000 substitutions to complete.

I'm not sure of the limitations on the -e flag...i.e. I have no idea howmany -e's you can have..but this may be high...(although I doubt it would be).

If you knew perl you could compile the similar with one pass of the file...although somewhat more effort to set up.