Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

sed command question

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed command question
# 1  
Old 06-25-2013
sed command question
Hey all,

so I've been experimenting with SED today, no experience before today, so if you're not patient, stop reading now! :P

I will attempt to explain this as simply as possible, without having to post massive walls of shitty code. Basically, I've created a small sed script to go through an XML document and append lines of XML after certain other lines. However, I've run into a slight glitch when I tried to nest an awk statement inside a sed statement.

My sed script is
Code:
sed '/KWName/a\
'`awk -F, -f sed.awk sednumbers`'' sedtest.txt

This awk script generates the following output:

Code:
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>

, which is what I want appended (and no, it's not nice and static, this is just for testing purposes). The awk script executes fine and generates the right output, but then at the top of my output I get

Code:
Can't open name="KWValue1">20*</entry>
Can't open <entry
Can't open name="KWValue2">21*</entry>
Can't open <entry
Can't open name="KWValue3">22*</entry>
Can't open <entry
Can't open name="KWValue4">23*</entry>
Can't open <entry
Can't open name="KWValue5">24*</entry>

and it only inserts the first <entry as the newline. Clearly it is reading the spaces as meaning this is a seperator between two files. How can I correct that?

For posterity's sake, the output I'm getting is
Code:
Can't open name="KWValue1">20*</entry>
Can't open <entry
Can't open name="KWValue2">21*</entry>
Can't open <entry
Can't open name="KWValue3">22*</entry>
Can't open <entry
Can't open name="KWValue4">23*</entry>
Can't open <entry
Can't open name="KWValue5">24*</entry>
<?xml version="1.0" encoding="utf-8">
<OBExport>
<section name="Query1">
<entry name="DocumentType">Document Type</entry>
<entry name="KWName1">Project Number</entry>
<entry
<entry name="KWName2">Org ID</entry>
<entry
<entry name="KWName3">Invoice Number</entry>
<entry
</section>
<section name="Query2">
<entry name="DocumentType">Invoices</entry>
<entry name="KWName1">Project Number</entry>
<entry
<entry name="KWName2">Org ID</entry>
<entry
<entry name="KWName3">Invoice Number</entry>
<entry
</section>
<section name="Query3">
<entry name="DocumentType">Requisitions</entry>
<entry name="KWName1">Invoice Number</entry>
<entry
</section>
<section name="Query4">
<entry name="DocumentType">Proposals</entry>
<entry name="KWName1">Project Number</entry>
<entry
<entry name="KWName2">Org ID</entry>
<entry
</section>
</OBExport>

and the output I want is

Code:
<?xml version="1.0" encoding="utf-8">
<OBExport>
<section name="Query1">
<entry name="DocumentType">Document Type</entry>
<entry name="KWName1">Project Number</entry>
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>
<entry name="KWName2">Org ID</entry>
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>
<entry name="KWName3">Invoice Number</entry>
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>
</section>
<section name="Query2">
<entry name="DocumentType">Invoices</entry>
<entry name="KWName1">Project Number</entry>
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>
<entry name="KWName2">Org ID</entry>
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>
<entry name="KWName3">Invoice Number</entry>
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>
</section>
<section name="Query3">
<entry name="DocumentType">Requisitions</entry>
<entry name="KWName1">Invoice Number</entry>
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>
</section>
<section name="Query4">
<entry name="DocumentType">Proposals</entry>
<entry name="KWName1">Project Number</entry>
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>
<entry name="KWName2">Org ID</entry>
<entry name="KWValue1">20*</entry>
<entry name="KWValue2">21*</entry>
<entry name="KWValue3">22*</entry>
<entry name="KWValue4">23*</entry>
<entry name="KWValue5">24*</entry>
</section>
</OBExport>

Thanks in advance for your time.
# 2  
Old 06-25-2013
Hi, this section..
Code:
`awk -F, -f sed.awk sednumbers`

is unprotected by double quotes, so the shell will interpret the < and > signs as redirects from / to files
# 3  
Old 06-25-2013
Hmm. Can you explain a little more how I would protect it with double quotes? The three ways I tried gave me:

Code:
$ sed '/KWName/a\
> '`"awk -F, -f sed.awk sednumbers"`'' sedtest.txt
ksh: awk -F, -f sed.awk sednumbers:  not found

and

Code:
$ sed "/KWName/a\
> "`awk -F, -f sed.awk sednumbers`"" sedtest.txt
sed: command garbled: /KWName/a<entry

and

Code:
$ sed '/KWName/a\
> '"`awk -F, -f sed.awk sednumbers`"'' sedtest.txt
Unrecognized command: <entry name="KWValue2">21*</entry>

Or is there a way I can do it without requiring the `? That last one seems to be closer to what I'm looking for, at least.
# 4  
Old 06-25-2013
Quote:
Originally Posted by Scrutinizer
Hi, this section..
Code:
`awk -F, -f sed.awk sednumbers`

is unprotected by double quotes, so the shell will interpret the < and > signs as redirects from / to files
I think your analysis is incorrect. The shell scans for redirection operators before any expansions occur.

It looks to me like field splitting ends the sed script immediately following the first word of the command substitution, <entry. Everything that follows in the command substitution is passed to sed as separate arguments which sed treats as filenames. I believe this is the source of the error messages (I don't have a shell handy to confirm).



Quote:
Originally Posted by Parrakarry
Hmm. Can you explain a little more how I would protect it with double quotes? The three ways I tried gave me:

Code:
$ sed '/KWName/a\
> '"`awk -F, -f sed.awk sednumbers`"'' sedtest.txt
Unrecognized command: <entry name="KWValue2">21*</entry>

Or is there a way I can do it without requiring the `? That last one seems to be closer to what I'm looking for, at least.
That is the correct way to do what you're trying to do. The problem is that newlines within the text argument to sed's a command must be backslash escaped. Since they're not, sed considers the text to end at the first newline. sed then tries to parse the second line of awk output as a sed command.

Regards,
Alister
This User Gave Thanks to alister For This Post:
Scrutinizer
# 5  
Old 06-25-2013
So is there a way to get awk to backslash escape the newline? Or am I just barking up the wrong tree?

sed.awk:

Code:
{
        x=1
        for ( i = 1; i <= NF; i++ )
        {
                if ( $i )
        {
                print "<entry name=\"KWValue"x"\">"$i"</entry>"
                x=x+1
        }
        }
}

And I just tried the obvious thing, which was adding a \\ to the sed.awk, causing the xml generated to look like

Code:
<entry name="KWValue1">20*</entry>\
<entry name="KWValue2">21*</entry>\
<entry name="KWValue3">22*</entry>\
<entry name="KWValue4">23*</entry>\
<entry name="KWValue5">24*</entry>\

Got a rather strange output from the sed command:

Code:
$ sed '/KWName/a\
> '"`awk -F, -f sed.awk sednumbers`"'' sedtest.txt
<?xml version="1.0" encoding="utf-8">
<OBExport>
<section name="Query1">
<entry name="DocumentType">Document Type</entry>
<entry name="KWName1">Project Number</entry>
<entry name="KWName2">Org ID</entry>
<entry name="KWName3">Invoice Number</entry>
</section>
<section name="Query2">
<entry name="DocumentType">Invoices</entry>
<entry name="KWName1">Project Number</entry>
<entry name="KWName2">Org ID</entry>
<entry name="KWName3">Invoice Number</entry>
</section>
<section name="Query3">
<entry name="DocumentType">Requisitions</entry>
<entry name="KWName1">Invoice Number</entry>
</section>
<section name="Query4">
<entry name="DocumentType">Proposals</entry>
<entry name="KWName1">Project Number</entry>
<entry name="KWName2">Org ID</entry>
</section>
</OBExport>

No errors, but it also did nothing to my input.
Last edited by Parrakarry; 06-25-2013 at 04:55 PM..
# 6  
Old 06-25-2013
Quote:
Originally Posted by alister
I think your analysis is incorrect. The shell scans for redirection operators before any expansions occur.

It looks to me like field splitting ends the sed script immediately following the first word of the command substitution, <entry. Everything that follows in the command substitution is passed to sed as separate arguments which sed treats as filenames. I believe this is the source of the error messages (I don't have a shell handy to confirm).
He, he you are right, already had a gnawing feeling when I posted it Smilie.. At any rate the double quotes would be required ..
# 7  
Old 06-25-2013
Well the double quotes certainly helped me move forward to a new error message, so that's something right? :P
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Sed command usage question

How to work x in sed command? I know x command is swaps the contents of pattern space and hold space. But i am unable to understand it's working? (4 Replies)
Discussion started by: Vartika18
4 Replies

2. Shell Programming and Scripting

sed and awk giving error ./sample.sh: line 13: sed: command not found

Hi, I am running a script sample.sh in bash environment .In the script i am using sed and awk commands which when executed individually from terminal they are getting executed normally but when i give these sed and awk commands in the script it is giving the below errors :- ./sample.sh: line... (12 Replies)
Discussion started by: satishmallidi
12 Replies

3. Shell Programming and Scripting

after sed command file shows '????' question marks

Hi All, I am trying to replace a string in a oracle LDT file file extension .ldt. I am using the following command: sed "s/Test Number Viewer/Test Number 1/g" TEST_LIEN_FORM.ldt > TEST_LIEN_FORM.ldt.tmp mv TEST_LIEN_FORM.ldt.tmp TEST_LIEN_FORM.ldt But after the command, the file... (2 Replies)
Discussion started by: veena484
2 Replies

4. Shell Programming and Scripting

Sed command garbled question

for j in $(cat ${list_B}) do to_replace_2=$(grep $j ${useralias}_2) sed "s/^${j}/${to_replace_2}/p" ${entries} > ${entries}_2 mv ${entries}_2 ${entries} done Hi, I've the above sed command running in a script. Its basically looping through a file and replacing its beginning of line... (8 Replies)
Discussion started by: Jazmania
8 Replies

5. Shell Programming and Scripting

Sed command question on Solaris

Hi, I'm trying to find the first field in a text file with the below sed command but it doesn't seem to be correct for running on Solaris.. It has no problem running on AIX. Anyone got a suggestion what the problem is? sed 's/^\(\+\) /OK/' The eventual goal is to separate the columns in a... (5 Replies)
Discussion started by: Jazmania
5 Replies

6. Shell Programming and Scripting

Question about a sed command

Hi guys, I'm currently trying to understand a piece of shell script and it has some sed commands. I've been looking through sed tutorials to figure out what it does but still no luck :confused: Can any of you guys tell me what this particular command does? sed -i '1i\.options' a/* ... (1 Reply)
Discussion started by: chu816
1 Replies

7. UNIX for Dummies Questions & Answers

Question on my sed command

So I have this sed command below. The content of the tmp.txt file is dv01:at01,at05,at02:at04 sed 's/\:.*\,/\,/g' tmp.txt Which produces dv01,at02:at04 and I'm trying to use sed to get me dv01,at05,at02 Stripping out the parts leading with ":". My sed is pretty basic, can... (5 Replies)
Discussion started by: J-Man
5 Replies

8. Shell Programming and Scripting

Loop with sed command to replace line with sed command in it

Okay, title is kind of confusion, but basically, I have a lot of scripts on a server that I need to replace a ps command, however, the new ps command I'm trying to replace the current one with pipes to sed at one point. So now I am attempting to create another script that replaces that line. ... (1 Reply)
Discussion started by: cbo0485
1 Replies

9. Shell Programming and Scripting

Sed Question 1. (Don't quite know how to use sed! Thanks)

Write a sed script to extract the year, rank, and stock for the most recent 10 years available in the file top10_mktval.csv, and output in the following format: ------------------------------ YEAR |RANK| STOCK ------------------------------ 2007 | 1 | Exxon... (1 Reply)
Discussion started by: beibeiatNY
1 Replies

10. Shell Programming and Scripting

Quick Question on sed command in shell script

Hello, I have the following line in one of my shell scripts. It works fine when the search string($SERACH_STR) exists in the logfile($ALERTLOG) but if the search string does not exist this line errors out at run time. Is there a way to make this line return 0 if it is not able to find the... (4 Replies)
Discussion started by: luft
4 Replies
Login or Register to Ask a Question