utf-8, problem with special characters


 
Thread Tools Search this Thread
Operating Systems HP-UX utf-8, problem with special characters
# 1  
Old 09-24-2008
utf-8, problem with special characters

Hi all,


We are facing the following problem in our HP-UX machine: software that manipulates utf-8 encoded strings (e.g. during string cut), fails to correctly manipulate strings (all containing Greek characters) that contain special characters like @, &, # etc. Actually, in different environments it behaves differently. In other unix environment it seems not having any problem but in fails in our environment which is

/usr/sbin/sizer -v
Compaq Tru64 UNIX V5.1A (Rev. 1885); Mon Apr 9 15:44:05 EEST 2007


Thanks in advance for any help
Alina
# 2  
Old 09-25-2008
This sounds like a locale issue. I understand what HPUX does, but I do not know much Tru64 and locale. If you review the localeconv, localedef, langinfo, and NLS man pages for HPUX it has information on how to construct a "locale". When you build a locale, one of the things that is defined is what characters are punctuation. Offhand I would guess the locales you are using has a definition problem. IF you need portabilty between your environments you have to find or define matching locale defintions for each environment

Actually not only cut but also every unix tool and library calls like strftime() will do odd things under a poorly set up locale.

Last edited by jim mcnamara; 09-25-2008 at 07:08 AM..
# 3  
Old 09-25-2008
The list of supported locales of Tru64 UNIX is documented in the l10n_intro.5 man page. The following Greek locales are supported on Tru64 UNIX V5.1A: el_GR.ISO8859-7
el_GR.ISO8859-7@ucs4 and el_GR.UTF-8.

What is the ouput when you run the locale utility i.e. 'locale'?

Do you see these locales when you run 'locale -a'?
# 4  
Old 09-26-2008
Hi All,

We have solved the problem by including the following line into user's .profile file:

export LANG=universal.UTF-8


After that, all software worked as expected. I think if we try also other codesets like the ones proposed by you, it would work too.

Many thanks for your help !
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Convert UTF-8 file to ASCII/ISO8859-1 OR replace characters

I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc 1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Discussion started by: hemkiran.s
3 Replies

2. Linux

Help to Convert file from UNIX UTF-8 to Windows UTF-16

Hi, I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt and i am getting some chinese characters as below which l opened the converted file on windows machine. LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies

3. Shell Programming and Scripting

Problem with special characters....

grep -i "$line,$opline" COMBO_JUNK|awk -F, ' { C4+=$4 } { } END { print C4 } ' OFS=,` when i run this command in the script.... it o/p all the value as 0 if $line contains any special parameters..... but the same script if i run in command prompt... it shows... (4 Replies)
Discussion started by: nikhil jain
4 Replies

4. Shell Programming and Scripting

Problem with Special characters in file

Hi, I am facing a below problem. Inorder to mak sure the below file is fixed width i am using the following command awk '{printf("%-375s\n", $0) } so as to add trailing spaces at the end for records of length less than 375. Input file > inp.txt 1©1234 1234 123©1 The output file is... (1 Reply)
Discussion started by: marcus_kosaman
1 Replies

5. Shell Programming and Scripting

script to tail file; problem with awk and special characters

Trying to use code that I found to send only new lines out of a log file by doing: while :; do temp=$(tail -1 logfile.out) awk "/$last/{p=1}p" logfile.out #pipe this to log analyzer program last="$temp" sleep 10 done Script works fine when logfile is basic text, but when it contains... (2 Replies)
Discussion started by: moo72moo
2 Replies

6. Shell Programming and Scripting

Replace special characters with Escape characters?

i need to replace the any special characters with escape characters like below. test!=123-> test\!\=123 !@#$%^&*()-= to be replaced by \!\@\#\$\%\^\&\*\(\)\-\= (8 Replies)
Discussion started by: laknar
8 Replies

7. Shell Programming and Scripting

special characters

Hey guys, I'm trying to replace "]Facebook" from the text but sed 's/]Facebook/Johan/g' is not working could you please help me with that? (6 Replies)
Discussion started by: Johanni
6 Replies

8. Shell Programming and Scripting

Special characters

When I open a file in vi, I see the following characters: \302\240 Can someone explain what these characters mean. Is it ASCII format? I need to trim those characters from a file. I am doing the following: tr -d '\302\240' ---------- Post updated at 08:35 PM ---------- Previous... (1 Reply)
Discussion started by: sid1982
1 Replies

9. Shell Programming and Scripting

special characters giving problem

Hi All, I have a CSV file in which some fields contains special character for ex:- my file is file 1 cat file1 abcd,bgfht,ngbht,abvc **** hdlld,hsgdt,bhfy,knht **** whenever i am trying to put a 4th feild in a variable its giving me list of all the files i have in current... (6 Replies)
Discussion started by: sam25
6 Replies

10. Shell Programming and Scripting

replace UTF-8 characters with tr

Hi, I try to get tr to replace multibytes characters by ascii equivalent. For example "Je vais ŕ l'école" ---> 'Je vais a l'ecole" But my version of tr (5.97) doesn't seem to support multibyte sets. $ locale charmap; echo "Je vais ŕ l'école" | tr éŕ ea UTF-8 Je vais aa l'aacole I try to... (2 Replies)
Discussion started by: ripat
2 Replies
Login or Register to Ask a Question