I'm going madder than I was this morning :) - Shell Programming and Scripting

Wolja · #1 (**permalink**) 06-03-2008

I'm creating a lot of test data for some performance testing coming up. The vendor product I used to create the file had a slight bug in it and got some times wrong so I decided to use sed to fix it.

This works on the command line but not in a shell script and I can't work out why.

File to be changed. I've added the $ sign as the termnal isn't wide enough.

I want to change 039914, for eg, to 100001 and say 039915 to 102000

so as expected sed ''s/039914/100001/'' file.dat > file1.dat works and I get what I want.

In the shell script , given I have about 2,000 substitutions to make , I cat a file with the substitution
eg
039914 100001
039915 102000

Then the script, again dead simple

Code:

 
#!/bin/sh 
rm b1.dat
touch b1.dat
j=`wc -l cnt1.txt|awk '{FS=" "};{print $1}'`

cnt=1
while [ $cnt -le $j ] ; do
   match=`sed -n "$cnt p" cnt1.txt |awk '{FS=" "};{print $1}'`
   replace=`sed -n "$cnt p" cnt1.txt |awk '{FS=" "};{print $2}'`
   `sed -e "s/$match/$replace/g" file.dat >> b1.dat` 
#(I've tried a bazillion different ways to do this line and even setenv at the #command prompt works)
   cnt=`expr $cnt + 1`
done

As expected I should be running sed "s/039914/100001/" at the bold line but for some reason it removes everything up to ANZ rather than substituting . A -xv gives a weird sed command which is what is confusing me.

eg
+ sed -n 1 p cnt1.txt
replace=100100
echo 039914 100100
echo "Replacing 039914 with 100100"
+
/g file.dat 9914/100100

Code:

if I use single quote , eg , sed -e 's/${match}/${replace}/g' file.dat I get the correct command echo but th evariables don't expand

Any thoughts?

Data sample.

Code:

"000000" "ZZ"$
"0000"$
"000001" "TA"$
"0" "1" "039914" "ANZ" "   " "01" "0" "2103" "7777" "022000000" "000001000" "000002200000" "0490" "20080528" "43133
3" "EP              " "        " "20080602" "          " "N" "          " "          " "000000000000"$
"000002" "TA"$
"0" "1" "039915" "CBA" "   " "01" "0" "2103" "7777" "047000000" "000011000" "000051700000" "0490" "20080528" "43134
6" "EP              " "        " "20080602" "          " "N" "          " "          " "000000000000"$
... for a half million rows.

zaxxon · #2 (**permalink**) 06-03-2008

If I got it right, you have problems with the variable substitution inside sed.
You can write it like this:

Code:

VAR="just"; echo "this is just a test" | sed 's/'$VAR'/not just/g'

Wolja · #3 (**permalink**) 06-03-2008

Quote:

Originally Posted by zaxxon

If I got it right, you have problems with the variable substitution inside sed.
You can write it like this:

Code:

VAR="just"; echo "this is just a test" | sed 's/'$VAR'/not just/g'

That wasn't quite the answer but got me thinking thanks

for some reason , not sure why, the backticks in the variable creation were interfering with the sed. Removing the backticks and suddenly it works.

I have no idea why though.
[code]
#!/bin/sh
rm b1.dat
touch b1.dat
j=`wc -l cnt1.txt|awk '{FS=" "};{print $1}'`

cnt=1
while [ $cnt -le $j ] ; do
match=`sed -n "$cnt p" cnt1.txt |awk '{FS=" "};{print $1}'`
replace=`sed -n "$cnt p" cnt1.txt |awk '{FS=" "};{print $2}'`
sed -e "s/$match/$replace/g" file.dat >> b1.dat
#(I've tried a bazillion different ways to do this line and even setenv at the #command prompt works)
cnt=`expr $cnt + 1`
done

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Hi Good Morning	ashokmeti	IP Networking	3	01-30-2008 02:42 PM
Pass the source, please - Sydney Morning Herald	iBot	UNIX and Linux RSS News	0	09-25-2007 05:40 AM
How do you schedule a command to run at 4:00 every morning?	JosephGerard	UNIX for Dummies Questions & Answers	1	07-21-2005 02:11 AM

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode