I am working on AIX. We ftp files to a database. The flat files are having thousands of records and each record is having some 50 to 60 characters(there are fields having certain character length). In addition to some valid ascii characters some invalid characters like Å, å, Ä, ä or pipes creep in which datawarehouse rejects to load in.
Example: AcuM-^?a 051706 ;
above is a field in the record which is having special characters like -,^ and ? , which should not have been there.

The record separator is a new line and there is no field seperator.

How can I remove these invalid or special characters to creep in the records?
Please help me to find the logic in the shell sripting..

Thanks for the reply Perderabo. There are some more doubts;

=>There can be lots more invalid character like these, so shall I give all them in the 'like' command?

=>If I want to replace the 'invalid character' with a 'space' how can I do that?

=>How to run whole process for thousands of records in the file?

Attached is an example in this regard...

Attached Files

Example.txt (1.4 KB, 3 views)

The
echo "invalid characters like Å, å, Ä, ä or"
is providing the input data with illegal characters that need removal. I need some test data and this is one way to demo a command. And the command I am showing is
tr -dc " a-zA-Z0-9,\n"
and that is what removes the garbage. The tr command, in this form, lists the valid characters, not the invalid ones. You may need to add stuff to the list. To replace invalid characters with a space use I have switched to single quotes which may be better if you need certain special characters to be accepted. In your case you may want to just do Read the tr man page for more info.

I used the translate commands as follows:


1) tr -c ' a-zA-Z0-9,\n' ' ' < inputfile > outputfile ==> No result

2) tr -c '[:print:][:cntrl:]' ' ' < inputfile > outputfile ==>

The second command replaced one of the invalid character with space but retained all others. When I again ran the command on the resulted file, the complete files.(input as well as output.)

Please tell me some other combinations.

===================================================
Among the following characters, ~æ£ÇÄ, Ç was replaced.
===================================================

Thanks in advance.
Kanu