Trouble with sed and substituting a string with special characters in variable


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Trouble with sed and substituting a string with special characters in variable
# 1  
Old 05-31-2012
Trouble with sed and substituting a string with special characters in variable

Hey guys,
I know that title is a mouthful - I'll try to better explain my struggles a little better...

What I'm trying to do is:

1. Query a db and output to a file, a list of column data.
2. Then, for each line in this file, repeat these values but wrap them with:

Code:
ITEM{
          LABEL = <LINEVALUEFROMFILE>
          VALUE = <LINEVALUEFROMFILE>
}

for each line in the file. I'm doing this and outputting that to a new file, and purging the old.
3. CAT the contents of the formatted output file, then substitute it into a .cfg file in place of a "Marker" or string value that is unique (in my case, a string called VENDORLIST.

I have everything working fine right up to the sed statement. I've tried using:
Code:
sed -i "s/$marker/$markerrepl/g" $formattedfile

but that returns:
Code:
sed: -e expression #1, char 60: unterminated `s' command

I've tried "protecting" the variables by single quoting them like this:

Code:
sed -i "s/'$marker'/'$markerrepl'/g" $formattedfile

but that returns a similiar error with a different character reported, and if I wrap the variables in "" it merely substitutes the marker string with the variable in double quotes (not the content of the variable).

Speaking of the variable content, at the end of the script I echo it out to screen so I know the content is there... all I can figure is there is an issue with characters within that variable and that's the problem...

By the way, here's an example of what that variable content looks like from the formatted file (step 2):
Code:
 
                                          ITEM{
                                                          LABEL = Panasonic Corporation of N.A.
                                                          VALUE = Panasonic Corporation of N.A.
                                          }
                                          ITEM{
                                                          LABEL = NewWave Technologies, Inc.
                                                          VALUE = NewWave Technologies, Inc.
                                          }
                                          ITEM{
                                                          LABEL = Kyocera Doc Solutions of America, Inc.
                                                          VALUE = Kyocera Doc Solutions of America, Inc.
                                          }
                                          ITEM{
                                                          LABEL = Ingram Micro, Inc.
                                                          VALUE = Ingram Micro, Inc.
                                          }

Are there any sed or awk experts out there can help me get this data substituted in for my marker string?
# 2  
Old 05-31-2012
You have told us everything - save for the contents of "$marker" and "$markerrepl", which would be the only thing necessary to know. How am i - how are we - supposed to give you advice regarding a string when we don't know the string?

I hope you aren't too dissatisfied with an answer which will be of only general nature:

Quote:
Originally Posted by ampsys
Code:
sed -i "s/$marker/$markerrepl/g" $formattedfile

When substituting from variable contents there are a few problems. The first is that this is prone to "injections" - a part of the search- or replacement-string will constitue a sed command itself. The output will probably not be what you had in mind. Example:

Code:
a="abc"
b="xyz"

sed "s/$a/$b/" /some/file       # will work as expected

a="abc/xyz"
sed "s/$a/$b/" /some/file       # will not work

A solution for this will be to escape all the characters special to sed prior to useing these strings. Escaping in sed is done with backslashes:

Code:
a="abc\/xyz"
b="xyz"

sed "s/$a/$b/" /some/file       # will work as expected


A second problem is shell-related: without quoting variables can break up lines into pieces. This can be partly overcome with proper quoting:

Code:
a="a b c"
b="x y"

sed s/$a/$b/ /some/file            # will not work
sed 's/'"$a"'/'"$b"'/g' /some/file # will work

This will not help you if variables contain line feeds.

I hope this helps.

bakunin

Last edited by bakunin; 06-01-2012 at 05:49 AM..
# 3  
Old 06-01-2012
Thanks for your reply, bakunin. Sorry it wasn't more clear, but in the OP, I was trying to convey that $marker was the string itself that I wanted to replace (VENDORLIST), and $markerrepl was the cat(ed) results of the formatted file I gave an example of:

Code:
                                                          ITEM{
                                                          LABEL = Panasonic Corporation of N.A.
                                                          VALUE = Panasonic Corporation of N.A.
                                                          }
                                                           ITEM{
                                                          LABEL = NewWave Technologies, Inc.
                                                          VALUE = NewWave Technologies, Inc.
                                                          }
                                                           ITEM{
                                                          LABEL = Kyocera Doc Solutions of America, Inc.
                                                          VALUE = Kyocera Doc Solutions of America, Inc.
                                                          }
                                                          ITEM{
                                                          LABEL = Ingram Micro, Inc.
                                                          VALUE = Ingram Micro, Inc.
                                                          }

Anyway, I guess we can close this. I gave up on sed and trying to figure out how to escape the special characters and just used perl to accomplish the desired result.

Thanks again for your help!
# 4  
Old 06-01-2012
Quote:
Originally Posted by ampsys
Thanks for your reply, bakunin. Sorry it wasn't more clear, but in the OP, I was trying to convey that $marker was the string itself that I wanted to replace (VENDORLIST), and $markerrepl was the cat(ed) results of the formatted file I gave an example of

Ok, sorry for not getting this earlier. So you search for a certain string and want to replace this string with the content of a file. This is relatively easy to do with the "r <file>" command, which reads a file.

As you said you already have a solution in PERL i won't go into details here, because it is non-trivial to do: You first have to split the line at the word searched for, put the part after it to hold space, delete the word itself, then read the file and append the content of the hold space back after. Finally put this whole procedure into a loop to cover for multiple occurrences of the search term.

bakunin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

getting error while substituting variable using sed..

I am using script for substitute one variable with another variable like below... below code works fine... sed 's/'$sub_fun'/'$To_sub'/g' But when i run the same code from the script getting below errors.. sed: -e expression #1, char 7: unterminated `s' command please help....... (2 Replies)
Discussion started by: pamu
2 Replies

2. Shell Programming and Scripting

Substituting a shell variable in sed

Hi guys, I'm trying to figure out how to use a shell variable inside my sed command. I just want to remove a certain part of a path. I've tried three different combinations and none of them work. Here are the three combinations: echo $file | sed 's/'$test'//' echo $file | sed "s/$test//"... (7 Replies)
Discussion started by: chu816
7 Replies

3. Shell Programming and Scripting

Sed failing for variable with special characters

This has been covered many times earlier but couldnt figure the issue myself. Can you please advise whats wrong on the below code I have a variable with special character ($) and am using that variable to replace another variable in file but however sed is failing to parse it correctly ... (7 Replies)
Discussion started by: sasiharitha
7 Replies

4. Programming

C++ Special Characters in a String?

Hello. How can i put all of the special characters on my keyboard into a string in c++ ? I tried this but it doesn't work. string characters("~`!@#$%^&*()_-+=|\}]{ How can i accomplish this? Thanks in advance. (1 Reply)
Discussion started by: cbreiny
1 Replies

5. Shell Programming and Scripting

Trouble cutting characters into a string.

I just have a couple of quick questions. I am having trouble with this cut. I am basically trying to cut the string so that i can insert the users guess at the appropriate point in the string. $letters is the character count of the $word. What it seems to do is cut the character into the... (0 Replies)
Discussion started by: Makaer
0 Replies

6. Shell Programming and Scripting

Special characters in a bash variable in sed

Hello, I am trying the following: echo __CHANGEME__ >> testfile VAR1="&&&" sed -i "s|__CHANGEME__|${VAR1}|" testfile cat testfile This results in testfile containing __CHANGEME____CHANGEME____CHANGEME__ Whereas I want it to result in &&& I understand that if VAR1="\&\&\&" then... (3 Replies)
Discussion started by: linuxnewbeee
3 Replies

7. Shell Programming and Scripting

Substituting Characters using SED

Can SED be used to substitute a character (y) with a character (Y) in a specified field? File has 12000 : delimeted rows as; HHC 1 BDE:Lastname, Firstname MI:firstname.mi.lastname@mil:SGT HHC 2 BDE:Lastname, Firstname MI:Firstname.MI.Lastname@mil:SGT I wish to replace the capital letters... (6 Replies)
Discussion started by: altamaha
6 Replies

8. Shell Programming and Scripting

Variable Manimpulation - special characters

Hello- I have a variables that contains a string like this usr/pass@SCHEMA I want to extract the usr/pass part and ignore the SCHEMA part, I tried to use this ${dbconn%%@} and apparently it will not work because @ is a special character. I tried \@ and still no go. Any idea how to solve... (1 Reply)
Discussion started by: Nomaad
1 Replies

9. Shell Programming and Scripting

Add string after another string with special characters

Hello everyone, I'm writing a script to add a string to an XML file, right after a specified string that only occurs once in the file. For testing purposes I created a file 'testfile' that looks like this: 1 2 3 4 5 6 6 7 8 9 And this is the script as far as I've managed: ... (2 Replies)
Discussion started by: heliode
2 Replies

10. Shell Programming and Scripting

Substituting with value of variable in Sed

Hi, I have a program in which i have to substitute a TAG in a file with the value of a variable. Code Snippet: ---------------- myvar=1234 sed 's/_TAG_/$myvar/' infile outfile When I run this command, the _TAG_ in the "infile" is substituted with "$myvar" but NOT the value "1234"... (1 Reply)
Discussion started by: jyotipg
1 Replies
Login or Register to Ask a Question