Shell Programming and Scripting

I have tried
sed -n 's/<w c5="\(.*\)" hw="\(.*\)" pos="\(.*\)">\(.*\)<\/w>/\1:\4/gp' A00.xml < test.txt

as you suggested. yet the output file of test.txt is still a mess. Here I attach the original file and the output file.

Is it that we haven't grepped all the contents in the pattern of <w c5="\(.*\)" hw="\(.*\)" pos="\(.*\)">\(.*\)</w> in a list form.

I try egrep. yet not working

Thanks for your discussion and instruction

This sounds like an assignment or homework, which is against the forum rules.

The problem is that ".*" does a greedy match, and you have multiple matches on each line of data, so you need to handle that. Try something like \"\([^"]*\)\" instead of "\(.*\)" to limit the match to the contents of the speech marks. [^"]* means any number of characters excluding ".

You will need some additional search and replaces to remove the <s ...> and <c ...> </c> tags, but I'll leave that as an exercise for you.

Thanks for your reply.

First, this is not homework or assignment. I am researching on corpus linguistics and try to find an effective way for collecting data. To remember items with parentheses are seldom mention in many examples.

I am afraid your version sed -n 's/<w c5="\(.*\)" hw="\(.*\)" pos="\(.*\)">\(.*\)<\/w>/\1:\4/gp' inputfile > outputfile doesn't work.

First, with/gp , the items in my results are repeated or doubled.

Second, I try sed 's/<w c5="\(.*\)" hw="\(.*\)" pos="\(.*\)">\(.*\)<\/w>/\1:\4/' test2.txt. Then it works. The content of test2.txt is like:
<w c5="VBZ" hw="be" pos="VERB">is</w>
<w c5="AT0" hw="a" pos="ART">a</w>
<w c5="NN1" hw="condition" pos="SUBST">condition</w>
<w c5="VVN" hw="cause" pos="VERB">caused</w>
<w c5="PRP" hw="by" pos="PREP">by</w>
<w c5="AT0" hw="a" pos="ART">a</w>
<w c5="NN1" hw="virus" pos="SUBST">virus</w>
<w c5="VVN" hw="call" pos="VERB">called</w>
<w c5="NP0" hw="hiv" pos="SUBST">HIV</w>
Then the result is

VBZ:is
AT0:a
NN1:condition
VVN:caused
PRP:by
AT0:a
NN1:virus
VVN:called
NP0:HIV

That means the sed only works for worklis like the above words in red part. Moreover, if we use "need some additional search and replaces to remove the *<s ...>* and *<c ...> </c>* tags", this may not be the best way.

I don't why it won't work for my whole file A00.xml

Best
John

I try to collect first those content like <w c5=".*" hw=".*" pos=".*?">.*</w> in that A00.xml.

I use the following pattern :

egrep "<w c5=".*" hw=".*" pos=".*?">.*</w>" A00.xml

The result is:

<s n="396"><w c5="PNP" hw="we" pos="PRON">We </w><w c5="VVB" hw="make" pos="VERB">make </w><w c5="AT0" hw="the" pos="ART">the </w><w c5="DT0" hw="most" pos="ADJ">most </w><w c5="PRF" hw="of" pos="PREP">of </w></s>

First, there is unexpected part <s n=...>

Second, they are not in list form like this:
<w c5="PNP" hw="we" pos="PRON">We </w>
<w c5="VVB" hw="make" pos="VERB">make </w>
<w c5="AT0" hw="the" pos="ART">the </w>
<w c5="DT0" hw="most" pos="ADJ">most </w>
<w c5="PRF" hw="of" pos="PREP">of </w>

I'm glad it's not homework, I just though I should check because we get a lot of posts like that here.

You don't seem to have tried what I suggested in my previous post to prevent greedy matching?

Is there any particular reason why you want to use sed? This perl one-liner seems to do what you require, as I understand it anyway:

Code:

perl -ne 'while (/<w c5="(.*?)" hw=".*?" pos=".*?">(.*?)<\/w>/g) {print $1:$2\n"}' A00.xml > outputfile

First, we have unix system installed in a server. we have many xml files as big as 4 G to process. Then I think the server can process them much faster than my desktop computer. Second, I 'v e not learned perl before and am afraid that it will assump too much of my time to learn a new script language. Third, I try other GNU softwares such as powergrep and textpipe. Yet they take money to buy after evaluation period. As far as my understanding, they offer similar functions for extract data according to regular expression. Then I want to make full use of the unix tool sed, awk , and grep to reach teh same functions like what these program do.

I don't know why you're telling me about your Unix system... I assumed you were doing this on Unix anyway? perl is widely found on Unix systems, and is more efficient at processing large amounts of data, so I would say it is ideal for your purposes (and very useful to learn!).

sed, awk and grep can also be used equally well for your task; I've given you some tips which you don't appear to have tried yet... so I'll wait until you give them a go. Let me know if you get stuck and have any specific questions.