Hello guys,

I'm cleaning out big XML files (we're talking about 1GB at least), most of them contain words written in a non-latin alphabet.

The command I'm using is so slow it's not even funny:

cat $1 | sed -e :a -e 's/&lt;[^&gt;]*&gt;//g;/&lt;/N;//ba;s/</ /g;s/>/ /g;s/_//g;s/-//g;s/–//g;s/(//g;s/)//g;s/,//g' | tr " " "\n" | sort | uniq >

I don't get why you would need a unique sort and the sed substitutions are not removing non-latin characters. Can you give a sample of input and desired output?

For clarity i decompose your sed-script first:

cat $1 |\
sed -e :a -e 's/<[^>]*>//g
                 /</N
                 //ba
                 s/</ /g
                 s/>/ /g
                 s/_//g
                 s/-//g
                 s/–//g
                 s/(//g
                 s/)//g
                 s/,//g' |\
tr " " "\n" |\
sort |\
uniq >

cat <file> | sed ...

sed '<commands>' <file>

sed 's/<spc>/<ENTER>/g' infile > outfile

<stream> | sort | uniq

<stream> | sort -u

sed 's/[<>]/ /g; s/[_-–(),]//g'

To insert a newline with sed's s command, the newline needs to be preceded by a backslash.

In this case, a simple, global substitution of space with newline, it's easier to use the y command, which supports the \n sequence:
y/ /\n/

A small nit: ^M is a carriage return character. ^J is newline/linefeed

Regards,
Alister

Thanks bakunin, it definitely helped. The syntax errors you pointed out were errors indeed. That's pretty much what made the whole thing not work as expected.


It's still a bit slow, but I can live with that

Thanks for your explanations.

Btw: I did not manage to remove / with sed, even when escaping the character... I had to use tr instead.