Shell Programming and Scripting

Hi,
I have many text files which contain some non-ASCII characters. I attach the screenshots of one of the files for people to have a look at. The issue is even after issuing the non-ASCII removal commands one of the characters does not go away. The character that goes away is the black one with a question mark in it whereas, the square character does not go. I also show how that square character looks when seen using the "more" command using konsole on Centos distribution.

The code which helps remove the black symbol with a question mark is this:

Code:

ls -1 *.dat | while read page
do
cat $page | tr -d '\001'-'\011''\013''\014''\016'-'\037''\200'-'\377' < "$page" > "$page".txt
done

But the above code does not get rid of those square characters. I have also searched through this forum and also on the internet and found many other codes to non-ASCII removal. None of them get rid of this square like character from the file.

Run octal dump and check what these characters are:

Code:

od -c < file

If these are control characters:

Code:

awk '{gsub(/[[:cntrl:]]/,X)}1' file > out

Thanks.

Code:

awk '{gsub(/[[:cntrl:]]/,X)}1' file > out

The above command helps get ride of those square ones. So this means by running this command along with the one that I have posted above can help get rid of all the non-ASCII characters. For od, I will study the output. This is something new for me.