Trouble with sed and substituting a string with special characters in variable
Hey guys,
I know that title is a mouthful - I'll try to better explain my struggles a little better...
What I'm trying to do is:
1. Query a db and output to a file, a list of column data.
2. Then, for each line in this file, repeat these values but wrap them with:
Code:
ITEM{
LABEL = <LINEVALUEFROMFILE>
VALUE = <LINEVALUEFROMFILE>
}
for each line in the file. I'm doing this and outputting that to a new file, and purging the old.
3. CAT the contents of the formatted output file, then substitute it into a .cfg file in place of a "Marker" or string value that is unique (in my case, a string called VENDORLIST.
I have everything working fine right up to the sed statement. I've tried using:
I've tried "protecting" the variables by single quoting them like this:
Code:
sed -i "s/'$marker'/'$markerrepl'/g" $formattedfile
but that returns a similiar error with a different character reported, and if I wrap the variables in "" it merely substitutes the marker string with the variable in double quotes (not the content of the variable).
Speaking of the variable content, at the end of the script I echo it out to screen so I know the content is there... all I can figure is there is an issue with characters within that variable and that's the problem...
By the way, here's an example of what that variable content looks like from the formatted file (step 2):
Code:
ITEM{
LABEL = Panasonic Corporation of N.A.
VALUE = Panasonic Corporation of N.A.
}
ITEM{
LABEL = NewWave Technologies, Inc.
VALUE = NewWave Technologies, Inc.
}
ITEM{
LABEL = Kyocera Doc Solutions of America, Inc.
VALUE = Kyocera Doc Solutions of America, Inc.
}
ITEM{
LABEL = Ingram Micro, Inc.
VALUE = Ingram Micro, Inc.
}
Are there any sed or awk experts out there can help me get this data substituted in for my marker string?
You have told us everything - save for the contents of "$marker" and "$markerrepl", which would be the only thing necessary to know. How am i - how are we - supposed to give you advice regarding a string when we don't know the string?
I hope you aren't too dissatisfied with an answer which will be of only general nature:
Quote:
Originally Posted by ampsys
Code:
sed -i "s/$marker/$markerrepl/g" $formattedfile
When substituting from variable contents there are a few problems. The first is that this is prone to "injections" - a part of the search- or replacement-string will constitue a sed command itself. The output will probably not be what you had in mind. Example:
Code:
a="abc"
b="xyz"
sed "s/$a/$b/" /some/file # will work as expected
a="abc/xyz"
sed "s/$a/$b/" /some/file # will not work
A solution for this will be to escape all the characters special to sed prior to useing these strings. Escaping in sed is done with backslashes:
Code:
a="abc\/xyz"
b="xyz"
sed "s/$a/$b/" /some/file # will work as expected
A second problem is shell-related: without quoting variables can break up lines into pieces. This can be partly overcome with proper quoting:
Code:
a="a b c"
b="x y"
sed s/$a/$b/ /some/file # will not work
sed 's/'"$a"'/'"$b"'/g' /some/file # will work
This will not help you if variables contain line feeds.
Thanks for your reply, bakunin. Sorry it wasn't more clear, but in the OP, I was trying to convey that $marker was the string itself that I wanted to replace (VENDORLIST), and $markerrepl was the cat(ed) results of the formatted file I gave an example of:
Code:
ITEM{
LABEL = Panasonic Corporation of N.A.
VALUE = Panasonic Corporation of N.A.
}
ITEM{
LABEL = NewWave Technologies, Inc.
VALUE = NewWave Technologies, Inc.
}
ITEM{
LABEL = Kyocera Doc Solutions of America, Inc.
VALUE = Kyocera Doc Solutions of America, Inc.
}
ITEM{
LABEL = Ingram Micro, Inc.
VALUE = Ingram Micro, Inc.
}
Anyway, I guess we can close this. I gave up on sed and trying to figure out how to escape the special characters and just used perl to accomplish the desired result.
Thanks for your reply, bakunin. Sorry it wasn't more clear, but in the OP, I was trying to convey that $marker was the string itself that I wanted to replace (VENDORLIST), and $markerrepl was the cat(ed) results of the formatted file I gave an example of
Ok, sorry for not getting this earlier. So you search for a certain string and want to replace this string with the content of a file. This is relatively easy to do with the "r <file>" command, which reads a file.
As you said you already have a solution in PERL i won't go into details here, because it is non-trivial to do: You first have to split the line at the word searched for, put the part after it to hold space, delete the word itself, then read the file and append the content of the hold space back after. Finally put this whole procedure into a loop to cover for multiple occurrences of the search term.
I am using script for substitute one variable with another variable like below...
below code works fine...
sed 's/'$sub_fun'/'$To_sub'/g'
But when i run the same code from the script getting below errors..
sed: -e expression #1, char 7: unterminated `s' command
please help....... (2 Replies)
Hi guys,
I'm trying to figure out how to use a shell variable inside my sed command.
I just want to remove a certain part of a path. I've tried three different combinations and none of them work. Here are the three combinations:
echo $file | sed 's/'$test'//'
echo $file | sed "s/$test//"... (7 Replies)
This has been covered many times earlier but couldnt figure the issue myself. Can you please advise whats wrong on the below code
I have a variable with special character ($) and am using that variable to replace another variable in file but however sed is failing to parse it correctly
... (7 Replies)
Hello. How can i put all of the special characters on my keyboard into a string in c++ ?
I tried this but it doesn't work.
string characters("~`!@#$%^&*()_-+=|\}]{
How can i accomplish this?
Thanks in advance. (1 Reply)
I just have a couple of quick questions.
I am having trouble with this cut. I am basically trying to cut the string so that i can insert the users guess at the appropriate point in the string.
$letters is the character count of the $word.
What it seems to do is cut the character into the... (0 Replies)
Hello,
I am trying the following:
echo __CHANGEME__ >> testfile
VAR1="&&&"
sed -i "s|__CHANGEME__|${VAR1}|" testfile
cat testfile
This results in testfile containing
__CHANGEME____CHANGEME____CHANGEME__
Whereas I want it to result in
&&&
I understand that if
VAR1="\&\&\&"
then... (3 Replies)
Can SED be used to substitute a character (y) with a character (Y) in a specified field?
File has 12000 : delimeted rows as;
HHC 1 BDE:Lastname, Firstname MI:firstname.mi.lastname@mil:SGT
HHC 2 BDE:Lastname, Firstname MI:Firstname.MI.Lastname@mil:SGT
I wish to replace the capital letters... (6 Replies)
Hello-
I have a variables that contains a string like this usr/pass@SCHEMA
I want to extract the usr/pass part and ignore the SCHEMA part, I tried to use this ${dbconn%%@} and apparently it will not work because @ is a special character. I tried \@ and still no go.
Any idea how to solve... (1 Reply)
Hello everyone,
I'm writing a script to add a string to an XML file, right after a specified string that only occurs once in the file. For testing purposes I created a file 'testfile' that looks like this:
1
2
3
4
5
6
6
7
8
9
And this is the script as far as I've managed:
... (2 Replies)
Hi,
I have a program in which i have to substitute a TAG in a file with the value of a variable.
Code Snippet:
----------------
myvar=1234
sed 's/_TAG_/$myvar/' infile outfile
When I run this command, the _TAG_ in the "infile" is substituted with "$myvar" but NOT the value "1234"... (1 Reply)
