Sponsored Content
Top Forums Shell Programming and Scripting Help replacing or scrubbing unicode characters Post 302160200 by adderek on Monday 21st of January 2008 06:30:09 AM
Old 01-21-2008
Do I understand it correctly that you want to frequently get rid of this character from some *.csv files? In my opinion best would be a simple C/C++ application that reads the file byte-by-byte and writes it byte-by-byte with skipping the unwanted ones. To improve performance you could optimize reads/writes.
 

10 More Discussions You Might Find Interesting

1. Programming

How to display unicode characters / unicode string

I have a stream of characters like "\u8BBE\u5907\u7BA1" and i want to display it. I tried following things already without any luck. 1) printf("%s",L("\u8BBE\u5907\u7BA1")); 2) printf("%lc",0x8BBE); 3) setlocale followed by fwide followed by wprintf 4) also changed the local manually... (3 Replies)
Discussion started by: jackdorso
3 Replies

2. AIX

problem with Unicode characters insertion

hi, I have a problem with unicode chars ( chinese, japanese etc ) insertion using sqlplus prompt. When i wrote a proc program for it i am able to create records. But when i fore the same query on sql prompt it stores reverse ????? ..some junk. widechar columns are mapped with NVARCHAR datatype.... (0 Replies)
Discussion started by: suman_jakkula
0 Replies

3. UNIX for Dummies Questions & Answers

replacing characters

Hi, I have a script for replacing bad characters in filenames for f in *; do mv $f `echo $f | tr '+' '_'` done; this replaces + for _ But I need to replace all bad characters ? / % + to _ Pls how can i do this in one script ? (3 Replies)
Discussion started by: palmer18
3 Replies

4. Shell Programming and Scripting

replacing characters

hi all I have a file that has sone spaces in start then / at last. i want to get rid of this. how to do? eg. 11414/ 49878/ 27627/ I WANT THE FILE AS 11414 49878 27627 PLEASE HELP (3 Replies)
Discussion started by: infyanurag
3 Replies

5. UNIX for Dummies Questions & Answers

remove special and unicode characters

Hi, How do I remove the lines where special characters or Unicode characters appear? The following query does work but I wonder if there is a better way. cat test.txt | egrep -v '\)|#|,|&|-|\(|\\|\/|\.' The following lines show that my query is incomplete. Warning: The word "*Khan" is... (1 Reply)
Discussion started by: shantanuo
1 Replies

6. Shell Programming and Scripting

Replacing characters

Hi fellow experts, I have a question for you. Data looks like: 00877,05/13/2010,PBO,P,0000708331,518 00877,05/13/2010,PBO,P,0000708331,519 ... ... 00877,05/13/2010,PBO,P,0000708331,2103 00877,05/13/2010,PBO,P,0000708331,2104,etc,etc Basically I have to replace 518,519,2103,2104,... (4 Replies)
Discussion started by: Devski123
4 Replies

7. Programming

How to make gl_get_line read unicode characters

Hi, My program uses gl_get_line from libtecla to get user input from terminal. It works fine as long as I enter English at the terminal prompt. However, if I enter other languages, such as Chinese characters, either by typing in or cut-and-paste, the input characters get cleared from terminal... (5 Replies)
Discussion started by: tdw
5 Replies

8. Shell Programming and Scripting

Perl script backspace not working for Unicode characters

Hello, My Perl script reads input from stdin and prints it out to stdout. After I read input I use BACKSPACE to erase characters. However BACKSPACE does not work with Unicode characters that are multi-bytes. On screen the character is erased but underneath only one byte is deleted instead of all... (3 Replies)
Discussion started by: tdw
3 Replies

9. Shell Programming and Scripting

sed replacing specific characters and control characters by escaping

sed -e "s// /g" old.txt > new.txt While I do know some control characters need to be escaped, can normal characters also be escaped and still work the same way? Basically I do not know all control characters that have a special meaning, for example, ?, ., % have a meaning and have to be escaped... (11 Replies)
Discussion started by: ijustneeda
11 Replies

10. Shell Programming and Scripting

Display unicode characters in zos shell

Hi all, I have a shell script that has several strings with \uxxxx characters distributed within. I would like to display these characters when I execute the script and echo the strings. I am running on zos in an sh environment. Some strings look like this: "Chcete-li pou\u017e\u00edt" <---... (1 Reply)
Discussion started by: adam.wis
1 Replies
orientation(5)							File Formats Manual						    orientation(5)

NAME
orientation - the orientation of a stream DESCRIPTION
The orientation of a stream is a property of a object that is handled as a input/output stream. It is useful when the input/output model assumes that characters are handled as wide-characters within an application and stored as multi-byte characters in files, and that all the wide-character input/output functions begin executing with the stream positioned at the boundary between two multi-byte characters. After a stream is associated with a file, but before any operations are performed on the stream, the stream is without orientation. If a wide-character input or output function is applied to a stream without orientation, the stream becomes wide-oriented implicitly. Likewise, if a byte input or output operation is applied to a stream without orientation, the stream becomes byte-oriented implicitly. Only the function can alter the orientation of a stream explicitly when the stream is without orientation. Just after a stream is associated with a pipe by the function, the stream is byte-oriented. After the stream becomes byte-oriented or wide-oriented, the orientation of a stream will be fixed and can not be changed until the stream is closed. The following functions are wide-character input/output functions. The following functions are byte input/output functions. EXAMPLES
To read characters from a stream when the orientation of the stream is unknown : WARNINGS
If byte input/output functions are applied to a wide-oriented stream or wide-character input/output functions are applied to a byte-ori- ented stream, the behavior is undefined. AUTHOR
The functionality of the orientation of a stream was developed by HP and Mitsubishi Electric Corporation. SEE ALSO
fgetws(3C), fopen(3S), fread(3S), fwide(3C), fwprintf(3C), fwscanf(3C), getc(3S), gets(3S), getwc(3C), popen(3S), printf(3S), putc(3S), puts(3S), putwc(3C), putws(3C), scanf(3S), ungetc(3S), ungetwc(3C). orientation(5)
All times are GMT -4. The time now is 07:38 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy