Shell Programming and Scripting

I have a file that has some accent characters in it when viewed in some text editors, but when viewed in vi they come in as ~R and ~U. I need to make a script to remove these characters from the file, but have been unsuccessful. I am not sure how sed or awk, or something similar is viewing them, so I am not sure what to tell it to replace. I have tried a few different options with no luck.

File I am working with is attached, some lines that contains invalid characters are: Line 99: “Schuckís Auto Parts JOB FAIR ...”, Line 882: “ìPoint of Viewî heart of Silverdale...”.

Please let me know if you have any suggestions. Thanks.

Hi,
I'mm on HP 11.11 and I can not see anything using vi?
A "cat -vetn testfile.txt | more" shows the character as M-^E and M-^R
Do a "man on cat" and read the description on "-v" talks about single bite control characters.
All we need to do is work out there ASCII code number and user a command like
# cat testfile.txt | tr -d "[ASCII CODES??]" > testfile2.txt

You need to do the following......
#Remove all the "GOOD" characters first
cat testfile.txt | tr -d '[:print"]" > badfile1

#Then find the ascii coodes of the bad characters
vis badfile1 | more

(Write down the ascii codes you see as you scroll through....)
(I found these codes in your file.....)
(\205 \222 \223 \224 \225)

#Remove the "bad" characters from testfile.txt
cat testfile.txt | tr -d "[\205\222\223\224\225]" > good_testfile.txt

Done!
NOTE: 1 You better look into what is causeing these "bad" characters to be enbedded in your file
NOTE: 2 Try and find out what the above ascii codes are!?

The above ascii codes where found in this particular file and in your next file there may be the same and/or different ones to search and remove
Good luck!

One line command to find the funny characters....
#cat testfile.txt | tr -d "[rint:]" | sort | vis

That way you will end up with a complete list of ascii codes that you need to remove

If I have time I'll try and work out how to just get a unique list of just the codes

This is awesome! I really appreciate the help.

I have a qestion, though. I ran through the process, and running the code using the ascii codes you found removed the characters I was having trouble with. However, when I ran this code:

cat testfile.txt | tr -d '[:print"]" > badfile1

to get ascii codes myself, all I got when I ran this:

vis badfile1 | more

was these characters: (no ascii codes)

\M^R & \M^E

Am I doing something wrong? I was expecting it to show me the ascii codes. Maybe I missed a step.

Nevermind. I appended the -o option to vis and it gave me the ascii codes. Thanks.

But, if you think of a way that we can only display unique ascii codes, that would be great.

Thanks again.

Ok-This is it....
To get a unique list of just the bad characters listed in your file try this....

cat testfile.txt | tr -d "[rint:][:space:]" | vis | fold -w 4 | sort -u

the "space" will remove the newline character so we end up with a single line of all the ascii codes. Then fold places them in a long multi line listing and sort only shows them once.

Good luck.... :-)