Shell Programming and Scripting

Hi Gurus,

I have file which contains some unicode charachator like "ü". I want to replace it with some charactors. I searched in internet and got command

Code:

 sed "s/ü/-/g"

, but I don't know how to type ü in unix command line.
Please help me for this one.

Thanks in advance

Be very careful, do not make an assumption that it is a single byte is size.

For a quick assessment use this command to check:-

Code:

hexdump -C /full/path/to/your/filename

This is an example; I copied your character and put it into an editor:-

Code:

This is the _byte_ ü _end_.

Note the character is between two spaces...

Now using the above command:-

Code:

Last login: Tue Jul  9 18:46:23 on ttys000
AMIGA:barrywalker~> hexdump -C /Users/barrywalker/byte_test.txt
00000000  54 68 69 73 20 69 73 20  74 68 65 20 5f 62 79 74  |This is the _byt|
00000010  65 5f 20 c3 bc 20 5f 65  6e 64 5f 2e 0a           |e_ .. _end_..|
0000001d
AMIGA:barrywalker~>

Note that at position 00000013 and 00000014 the bytes c3 and bc have appeared instead of the single character you are expecting...

So be very, very careful...

Hope this helps...

Thanks for your quick reply.
I run following command and got some result.

Code:

 # echo 'ADDÜL' |hexdump -C
00000000  41 44 44 dc 4c 0a                                 |ADD.L.|
00000006

Actually, I was run following command to split the file with one line to separate lines. when it hits the charactor Ü, it stopped.
what should I do to make the command to spearate file without stop

Code:

awk -v L="$2" '{for (i=1; i<=length($0); i+=L) print substr($0, i, L)}' "$1" > "$1"_split

Thanks in advance

I get a completely different result to you using the same command AFTER copying and pasting:-

Code:

Last login: Tue Jul  9 19:07:11 on ttys000
AMIGA:barrywalker~> echo 'ADDÜL' |hexdump -C
00000000  41 44 44 c3 9c 4c 0a                              |ADD..L.|
00000007
AMIGA:barrywalker~>

Notice I have 2 binary values, c3 and 9c, so it makes for a difficult cure...

However, IF, you can guarantee a single constant byte value you could try a derivative of this idea out:-

https://www.unix.com/shell-programmin...ipulation.html

I guess I don't understand the problem. You have created the command line you want to use above. Why can't you just copy it and paste it into your shell? Or copy it and paste it into a shell script using your editor?

The way you type unicode characters using a keyboard will vary depending on your operating system, your keyboard, and your current locale settings, but as long as your current locale and the character you're copying are both using the same underlying codeset, copy and paste should work.

Note that on most UNIX and Linux systems there won't be a locale that uses Unicode as the underlying codeset, but there are probably several that use UTF-8 (which is a multi-byte codeset that can encode any Unicode character).

Note that whether ü is a single-byte character (as it is in some EBCDIC code page variants and some ISO 8859-* codesets) or a multi-byte character (as it is in UTF-8) shouldn't matter to sed. The sed utility operates on characters; not bytes. You just need to be sure that the locale you're using when running sed is using a codeset with the same encoding for ü as the encoding used in the file you're editing.

Also in post 1 this character:-

Code:

ü

Is not the same as this character in post 3:-

Code:

Ü

So which is it?

This could be part of your problem...

It is

Code:

Ü