Sponsored Content
Top Forums Shell Programming and Scripting Removing these non-ASCII characters from a file Post 302828153 by shoaibjameel123 on Monday 1st of July 2013 11:18:24 PM
Old 07-02-2013
Removing these non-ASCII characters from a file

Hi,
I have many text files which contain some non-ASCII characters. I attach the screenshots of one of the files for people to have a look at. The issue is even after issuing the non-ASCII removal commands one of the characters does not go away. The character that goes away is the black one with a question mark in it whereas, the square character does not go. I also show how that square character looks when seen using the "more" command using konsole on Centos distribution.

The code which helps remove the black symbol with a question mark is this:
Code:
ls -1 *.dat | while read page
do
cat $page | tr -d '\001'-'\011''\013''\014''\016'-'\037''\200'-'\377' < "$page" > "$page".txt
done

But the above code does not get rid of those square characters. I have also searched through this forum and also on the internet and found many other codes to non-ASCII removal. None of them get rid of this square like character from the file.
Removing these non-ASCII characters from a file-screenpng
Removing these non-ASCII characters from a file-screen2png

Last edited by shoaibjameel123; 07-02-2013 at 12:20 AM.. Reason: Code tags edit
 

10 More Discussions You Might Find Interesting

1. HP-UX

Hex characters of ascii file

Hi, Whats the command or how do you display the hexadecimal characters of an ascii file. thanks Bud (2 Replies)
Discussion started by: budrito
2 Replies

2. Shell Programming and Scripting

Weird Ascii characters in file names

Hi. I have files in my OS that has weird file names with not-conventional ascii characters. I would like to run them but I can't refer them. I know the ascii # of the problematic characters. I can't change their name since it belongs to a 3rd party program... but I want to run it. is there... (2 Replies)
Discussion started by: yamsin789
2 Replies

3. Shell Programming and Scripting

Removing certain characters in a file

Hi I have a file that has semicolons in it (;) is there a way to just remove these in the file. Example name: Joe Smith; group: Group1; name: Mary White; group: Group2; (2 Replies)
Discussion started by: bombcan
2 Replies

4. Shell Programming and Scripting

convert ascii values into ascii characters

Hi gurus, I have a file in unix with ascii values. I need to convert all the ascii values in the file to ascii characters. File contains nearly 20000 records with ascii values. (10 Replies)
Discussion started by: sandeeppvk
10 Replies

5. Shell Programming and Scripting

Removing ^M characters from a file

Hi, I want to removing ^M characters from a file and combine the line with the next line. ex: issue i have: ABC^M^M DEF solution i need: ABCDEF I found that you by using the following command you can remove new line characters. tr -d '\r' < infile.csv > outfile.csv still... (10 Replies)
Discussion started by: mwrg
10 Replies

6. Shell Programming and Scripting

New line characters in Ascii file

I am having a file(1234.txt) downloaded from windows server (in Ascii format).However when i ftp this file to Unix server and try to work with it..i am unable to do anything.When i try to open the file using vi editor the file opens in the following format ... @ @ @ @ @ @ @ @... (4 Replies)
Discussion started by: appu2176
4 Replies

7. Shell Programming and Scripting

How to check if the file has EBCDIC or ascii characters

Hi, is there a way to check if the initial few characters are ebcdic or ascii in a file? (1 Reply)
Discussion started by: ahmedwaseem2000
1 Replies

8. Shell Programming and Scripting

Removing last and first characters in a file

bash-3.00$ cat temp.txt ./a/REA01/ces1/apps/ces_ces1_init3_aa.ear/ces.war/WEB-INF/classes/reds/common/environment.properties ./a/REA01/ces1/apps/ces_ces1_init3_aa.ear/commonproperties/hi/HostIntegration.properties... (9 Replies)
Discussion started by: bhas85
9 Replies

9. Shell Programming and Scripting

Identify extended ascii characters in a file

Hi, Is there a way to identify the lines in a file having extended ascii characters and display the same? For instance I have a file abc.txt having below data aaa|bbb|111|This is first line aaa|bbb|222|This is secõnd line aaa|bbb|333|This is third line aaa|bbb|444|This is foùrth line... (3 Replies)
Discussion started by: decci_7
3 Replies

10. Shell Programming and Scripting

Convert UTF-8 file to ASCII/ISO8859-1 OR replace characters

I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc 1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Discussion started by: hemkiran.s
3 Replies
tr(1B)						     SunOS/BSD Compatibility Package Commands						    tr(1B)

NAME
tr - translate characters SYNOPSIS
/usr/ucb/tr [-cds] [ string1 [string2]] DESCRIPTION
The tr utility copies the standard input to the standard output with substitution or deletion of selected characters. The arguments string1 and string2 are considered sets of characters. Any input character found in string1 is mapped into the character in the corresponding posi- tion within string2. When string2 is short, it is padded to the length of string1 by duplicating its last character. In either string the notation: a-b denotes a range of characters from a to b in increasing ASCII order. The character , followed by 1, 2 or 3 octal digits stands for the character whose ASCII code is given by those digits. As with the shell, the escape character , followed by any other character, escapes any special meaning for that character. OPTIONS
Any combination of the options -c, -d, or -s may be used: -c Complement the set of characters in string1 with respect to the universe of characters whose ASCII codes are 01 through 0377 octal. -d Delete all input characters in string1. -s Squeeze all strings of repeated output characters that are in string2 to single characters. EXAMPLES
Example 1: Creating a list of all the words in a filename The following example creates a list of all the words in filename1, one per line, in filename2, where a word is taken to be a maximal string of alphabetics. The second string is quoted to protect `' from the shell. 012 is the ASCII code for NEWLINE. example% tr -cs A-Za-z '12' <filename1>filename2 ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWscpu | +-----------------------------+-----------------------------+ SEE ALSO
ed(1), ascii(5), attributes(5) NOTES
Will not handle ASCII NUL in string1 or string2. tr always deletes NUL from input. SunOS 5.10 26 Sep 1992 tr(1B)
All times are GMT -4. The time now is 05:36 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy