Sponsored Content
Top Forums Shell Programming and Scripting Removing these non-ASCII characters from a file Post 302828153 by shoaibjameel123 on Monday 1st of July 2013 11:18:24 PM
Old 07-02-2013
Removing these non-ASCII characters from a file

Hi,
I have many text files which contain some non-ASCII characters. I attach the screenshots of one of the files for people to have a look at. The issue is even after issuing the non-ASCII removal commands one of the characters does not go away. The character that goes away is the black one with a question mark in it whereas, the square character does not go. I also show how that square character looks when seen using the "more" command using konsole on Centos distribution.

The code which helps remove the black symbol with a question mark is this:
Code:
ls -1 *.dat | while read page
do
cat $page | tr -d '\001'-'\011''\013''\014''\016'-'\037''\200'-'\377' < "$page" > "$page".txt
done

But the above code does not get rid of those square characters. I have also searched through this forum and also on the internet and found many other codes to non-ASCII removal. None of them get rid of this square like character from the file.
Removing these non-ASCII characters from a file-screenpng
Removing these non-ASCII characters from a file-screen2png

Last edited by shoaibjameel123; 07-02-2013 at 12:20 AM.. Reason: Code tags edit
 

10 More Discussions You Might Find Interesting

1. HP-UX

Hex characters of ascii file

Hi, Whats the command or how do you display the hexadecimal characters of an ascii file. thanks Bud (2 Replies)
Discussion started by: budrito
2 Replies

2. Shell Programming and Scripting

Weird Ascii characters in file names

Hi. I have files in my OS that has weird file names with not-conventional ascii characters. I would like to run them but I can't refer them. I know the ascii # of the problematic characters. I can't change their name since it belongs to a 3rd party program... but I want to run it. is there... (2 Replies)
Discussion started by: yamsin789
2 Replies

3. Shell Programming and Scripting

Removing certain characters in a file

Hi I have a file that has semicolons in it (;) is there a way to just remove these in the file. Example name: Joe Smith; group: Group1; name: Mary White; group: Group2; (2 Replies)
Discussion started by: bombcan
2 Replies

4. Shell Programming and Scripting

convert ascii values into ascii characters

Hi gurus, I have a file in unix with ascii values. I need to convert all the ascii values in the file to ascii characters. File contains nearly 20000 records with ascii values. (10 Replies)
Discussion started by: sandeeppvk
10 Replies

5. Shell Programming and Scripting

Removing ^M characters from a file

Hi, I want to removing ^M characters from a file and combine the line with the next line. ex: issue i have: ABC^M^M DEF solution i need: ABCDEF I found that you by using the following command you can remove new line characters. tr -d '\r' < infile.csv > outfile.csv still... (10 Replies)
Discussion started by: mwrg
10 Replies

6. Shell Programming and Scripting

New line characters in Ascii file

I am having a file(1234.txt) downloaded from windows server (in Ascii format).However when i ftp this file to Unix server and try to work with it..i am unable to do anything.When i try to open the file using vi editor the file opens in the following format ... @ @ @ @ @ @ @ @... (4 Replies)
Discussion started by: appu2176
4 Replies

7. Shell Programming and Scripting

How to check if the file has EBCDIC or ascii characters

Hi, is there a way to check if the initial few characters are ebcdic or ascii in a file? (1 Reply)
Discussion started by: ahmedwaseem2000
1 Replies

8. Shell Programming and Scripting

Removing last and first characters in a file

bash-3.00$ cat temp.txt ./a/REA01/ces1/apps/ces_ces1_init3_aa.ear/ces.war/WEB-INF/classes/reds/common/environment.properties ./a/REA01/ces1/apps/ces_ces1_init3_aa.ear/commonproperties/hi/HostIntegration.properties... (9 Replies)
Discussion started by: bhas85
9 Replies

9. Shell Programming and Scripting

Identify extended ascii characters in a file

Hi, Is there a way to identify the lines in a file having extended ascii characters and display the same? For instance I have a file abc.txt having below data aaa|bbb|111|This is first line aaa|bbb|222|This is secõnd line aaa|bbb|333|This is third line aaa|bbb|444|This is foùrth line... (3 Replies)
Discussion started by: decci_7
3 Replies

10. Shell Programming and Scripting

Convert UTF-8 file to ASCII/ISO8859-1 OR replace characters

I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc 1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Discussion started by: hemkiran.s
3 Replies
TRIM(3) 								 1								   TRIM(3)

trim - Strip whitespace (or other characters) from the beginning and end of a string

SYNOPSIS
string trim (string $str, [string $character_mask = " 0r B"]) DESCRIPTION
This function returns a string with whitespace stripped from the beginning and end of $str. Without the second parameter, trim(3) will strip these characters: o " " (ASCII 32 ( 0x20)), an ordinary space. o " " (ASCII 9 ( 0x09)), a tab. o " " (ASCII 10 ( 0x0A)), a new line (line feed). o " " (ASCII 13 ( 0x0D)), a carriage return. o "" (ASCII 0 ( 0x00)), the NUL-byte. o "x0B" (ASCII 11 ( 0x0B)), a vertical tab. PARAMETERS
o $str - The string that will be trimmed. o $character_mask - Optionally, the stripped characters can also be specified using the $character_mask parameter. Simply list all characters that you want to be stripped. With .. you can specify a range of characters. RETURN VALUES
The trimmed string. EXAMPLES
Example #1 Usage example of trim(3) <?php $text = " These are a few words :) ... "; $binary = "x09Example stringx0A"; $hello = "Hello World"; var_dump($text, $binary, $hello); print " "; $trimmed = trim($text); var_dump($trimmed); $trimmed = trim($text, " ."); var_dump($trimmed); $trimmed = trim($hello, "Hdle"); var_dump($trimmed); $trimmed = trim($hello, 'HdWr'); var_dump($trimmed); // trim the ASCII control characters at the beginning and end of $binary // (from 0 to 31 inclusive) $clean = trim($binary, "x00..x1F"); var_dump($clean); ?> The above example will output: string(32) " These are a few words :) ... " string(16) " Example string " string(11) "Hello World" string(28) "These are a few words :) ..." string(24) "These are a few words :)" string(5) "o Wor" string(9) "ello Worl" string(14) "Example string" Example #2 Trimming array values with trim(3) <?php function trim_value(&$value) { $value = trim($value); } $fruit = array('apple','banana ', ' cranberry '); var_dump($fruit); array_walk($fruit, 'trim_value'); var_dump($fruit); ?> The above example will output: array(3) { [0]=> string(5) "apple" [1]=> string(7) "banana " [2]=> string(11) " cranberry " } array(3) { [0]=> string(5) "apple" [1]=> string(6) "banana" [2]=> string(9) "cranberry" } NOTES
Note Possible gotcha: removing middle characters Because trim(3) trims characters from the beginning and end of a string, it may be confusing when characters are (or are not) removed from the middle. trim('abc', 'bad') removes both 'a' and 'b' because it trims 'a' thus moving 'b' to the beginning to also be trimmed. So, this is why it "works" whereas trim('abc', 'b') seemingly does not. SEE ALSO
ltrim(3), rtrim(3), str_replace(3). PHP Documentation Group TRIM(3)
All times are GMT -4. The time now is 02:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy