UNIX for Dummies Questions & Answers

Hi,

We are receiving an XML file in Unix which has some special characters between tags like '^' etc

<Tag> 1e^O7f%<2304e.$d8f57e8^Bf-&e.^Zh7/327e^O7 </Tag>

We need to remove all special characters like ^ ones and also any '&' or '<' or '>' being sent within the start and close tags i.e. in tag text.

The upstream system is sending some unicode characters which are getting convrted to carot symbols in Unix (apart from & and > and <). This is causing my XML parser to abort or drop rows which have such data.

Please provide a perl command to remove them. (we need to remove '&' and '<' and '>' which are present in tag 'text')

Thanks
DSR

Some things show out-of-range characters AS ^Z or whatever, but that doesn't mean it's literally the character ^ followed by the character Z. That's just its way of showing you characters it can't represent any other way.

So I think deleting the UTF8 characters themselves would be a good thing to try first; they're probably still there, unconverted. Since all UTF8 characters are >=128, we can use tr to strip out that entire range.

Code:

tr -d '[\200-\377]' < input > output

Thanks

I tried to run the below command but it didnt work, so I am assuming the XML file on Unix has infact chars in ascii

tr -d '[\200-\377]' < MyOld.xml > MyNew.xml

Can you post output of:

Code:

cat MyOld.xml | head

and

Code:

cat -e MyOld.xml | head

The UTF chars are still fine, I am more worried about the additional '&'/'<'/'>' which I have to romove from Tagtexts as they are failing my XML parser

---------- Post updated at 03:57 PM ---------- Previous update was at 03:53 PM ----------

Hi Bartus11, unfortunately my XML is a big single record. And your commands are pulling the whole XML. Please let me know what specific head info are you looking for

Would it be possible to re-download these XML files in an unconverted state? I think someone tried to remove the utf8 with cat -v and ruined it.

OK, so try this:

Code:

cut -b1-200 MyOld.xml | od -c

I hope there are some of those mysterious characters in the first 200 bytes of your file.