(Note that the $ at the end of the output is my shell's prompt. The arguments you are giving to tr are treated as octal values (not decimal), \145 is the character e; \147 is the character g; \128 is treated as \12 (the newline character) followed by the character 8; and 8-\140 in ASCII removes the characters 8, 9, all upper-case alphabetic characters, and the [, \, ], ^, _, and ` characters.
because, as I said before, the two byte character Š in UTF-8 is made up of bytes with the decimal values 197 and 160 (neither of which are in your list of byte values to be deleted by the sed command). (Note also that while, \dx (where x is a one, two, or three digit decimal number) works on some systems, it is an extension to the standards and, on many systems, will give you a syntax error or delete the characters d, 0, 1, 4, 5, 7, and 8.)
Please show us the output you get when you run the commands above!
I repeat:
What OS (including version) and shell are you using?
What Locale are you using when you run your script?
Last edited by Don Cragun; 01-02-2015 at 08:58 PM..
Reason: Fix typo (your s/b you).
korn shell (Version AJM 93t+ 2010-06) on Linux OS (2.6.18)
Apologies, I am unable to copy and paste results to and from the client network.
The "Š" was a character that I picked up to show how I wanted to remove such characters. It's good that I came to know new thing that we have multi byte characters as well.
I just noticed that I was not removing the characters properly as Don mentioned even with tr command.
In your last example, check the output of cat -v first; then you know where the dashes come from. And, not all systems/commands accept the \dnnn sequences. Try
In UTF-8 (and other UTFs), single chars above the ASCII range don't exist. They come in pairs or even longer char groups. So you could
- delete ALL chars above ASCII
- explicitly list the chars to be removed
- use iconv or recode to convert to e.g. "extended ASCII" (of which several char sets exist) and then remove those unwanted chars.
Example for option 2:
Code:
FN=$1
shift
TBD=$@
TBD=${TBD// /\|}
sed -r "s/$TBD//g" $FN
If you just want to get rid of non-ASCII characters (rather than a particular list of single- and/or multi-byte UTF-8 characters), the following awk and tr commands should work:
$ printf '%s\n' 'testing_Š_testing' 'testing__testing'| LANG=C awk '{gsub(/[\200-\377]/, "")}1'|od -c
0000000 t e s t i n g _ _ t e s t i n g
0000020 \n t e s t i n g _ _ t e s t i n
0000040 g \n
0000042
$ printf '%s\n' 'testing_Š_testing' 'testing__testing'| LANG=C tr -d '\200-\377'|od -c
0000000 t e s t i n g _ _ t e s t i n g
0000020 \n t e s t i n g _ _ t e s t i n
0000040 g \n
0000042
Hello,
I am on AIX.
When I encounter extended ascii characters and special characters on a file I need to print..
Byte position, actual character and line number.
Is there a simple command that can give me the above result ?
Thanks in advance (38 Replies)
I am working with a log file that I am trying to clean up by removing non-English ASCII characters. I am using Bash via Cygwin on Windows.
Before I start I set:
export LC_ALL=C
I clean it up by removing all non-English ASCII characters with the following command;
grep -v $''... (4 Replies)
Hi,
I want to read extended ASCII characters from keyboard using c language on unix/linux. How to read extended characters from keyboard or by copy-paste in terminal irrespective of locale set in the system. I want to read the input characters from keyboard, store it in an array or some local... (3 Replies)
We are getting extended Ascii characters in the input file and my requirement is to search and replace them with a space. I am using the following command
LANG=C sed -e 's// /g'
It is doing a good job, but in some cases it is replacing the extended characters with two spaces. So my input... (12 Replies)
Hi,
Is there a way to identify the lines in a file having extended ascii characters and display the same?
For instance I have a file abc.txt having below data
aaa|bbb|111|This is first line
aaa|bbb|222|This is secõnd line
aaa|bbb|333|This is third line
aaa|bbb|444|This is foùrth line... (3 Replies)
Hi all,
I have a file with extended ASCII codes in the description which needs to be removed.
List of extended ascii codes
"Œ", "œ", "Š", "š", "Ÿ", "ƒ", "-", "-", "‘",
"'", "‚", "“", "”", "„","†", "‡", "•",
"...", "‰", "€", "™"
Sample data:
Test Details-HAVE BEEN PUBLISHED... (1 Reply)
Hi All,
I'm trying to send extended ascii characters to my HP2055 as part of PCL printer control codes. What I want to do is select a bar code font, print the bar code and reset the printer to the default font.
Selecting the bar code font works good. Printing the bar code goes almost ok too. ... (5 Replies)
Hi, I have a accentuated letter (ö) in a script for an Installer. It's a file name. This is not working and I'm told to try using the octal value for the extended ascii character. Does anyone no how to do this? If I had the word "filförval", can I just put in the value between the letters, like... (9 Replies)
hi i would like to check text files if they contain extended ascii characters within or not. i really dont have any idea how to start your kind help would be very much appreciated thanks. (7 Replies)
Hi all,
I would like to change the extended ascii code ( 128 - 255).
I tried to change LC_ALL and LANG in current session ( values from locale -a) and for no good.
Thanks. (0 Replies)
