Shell Programming and Scripting

My file consist Ä which is an UTF-16 character , which can't be ident
ified by grep -P -n "[\x80-\xFF]".

And in the same file à is present which is an UTF-8 character and will be identified by grep -P -n "[\x80-\xFF]" .

One possibility that all the UTF-16 can be converted to UTF-8 if no regular expression is present. But it will take time to convert all of them.

Also not sure when it will convert UTF-16(2 byte char) to UTF-8(1 byte),how the alignment will be adjusted.

Looking for a way to find them .

Moderator's Comments:

CODE tags are to be used when displaying sample code segments, sample input, and sample output. HTML tags are to be used when displaying HTML code. Plain text describing requirements should not be tagged.

UTF8 or UTF16 are a character encodings that apply to an entire file, not a single character. So chances are minimal that in the same text a UTF16 Ä and a UTF8 Ã are present, at least if prepared by a reasonable application.

What be the output of file your_input_file

UTF-8 is not a 1 byte encoding, it uses 1 or more bytes depending on the character being encoded, which is why character replacement can't catch everything.

If your file is UTF-16, it ought to look very strange to normal UNIX programs because of the extra NULLs between most characters. Try hexdump -C to see what your file actually looks like.

The File contains both UTF-8(Ã) and (Ä) which can not be identified by using this range value."[\x80-\xFF]"

I need to filter all this Special character("[\x80-\xFF]" ) and other special character as well

It appears that you need to alter your locale while trying to process the file.

If you have - you are sure I gather - UTF8 characters then you need that locale to "see".
UNIX tools like awk understand locale settings.

The iconv tool can convert files from one locale to another - so this is another approach:
convert the file to match the locale you now run.

Hi All,

I am able to convert the UTF-16 characters to UTF-8 characters.

But when it is getting converted(using iconv) as UTF-16 is 2 byte value , hence the file alignment is getting changed. Is there any way to fix this to resolve the alignment problem.

My file contains only UTF-8 value which takes 1 byte. And each line it can have only limited number of UTF-8 chracters. But during conversion few of the UTF-8 characters are getting pushed to next line which is altering the file alignment.

I have tried to use recode command but that is not working. Any help would be greatly appreciated.

In general UNIX is and its utilities are coding-insensitive. That is: using sed (or awk, tr or similar text filters) you work on streams of bytes. In ASCII (and similar encodings) a "character" is a byte and a byte is a character. In other encodings this is not the case (like in UTF-16, where 2 bytes represent a character). But UNIX tools are not aware of this and treat each byte as if it would represent a character.

Having said this: to search for/find a text like "abc" regardless of the encoding cannot be done with these text filters, because they will not recognize that two bytes containing (if memory serves correctly) the hex values "00:61" ("U+0061") is the same letter "a" as a single byte with hex value "61" in ASCII.

Issuing grep "a" /some/file is basically telling grep to search for a byte containing the hex value 61 because this is what "a" is encoded in ASCII with.

Somebody has mentioned that python-tools work differently and are encoding-aware. It seems you need to resort to those (or similarily coding-aware tools) to get what you want. With UNIX tools you will not get over the aforementioned limitation, however clever you may work around that. It will always be a less-than-genuine solution which oughts to break under some unexpected set of conditions.

I hope this helps.

bakunin