I am working on AIX. We ftp files to a database. The flat files are having thousands of records and each record is having some 50 to 60 characters(there are fields having certain character length). In addition to some valid ascii characters some invalid characters like Å, å, Ä, ä or pipes creep in which datawarehouse rejects to load in.
Example: AcuM-^?a 051706 ;
above is a field in the record which is having special characters like -,^ and ? , which should not have been there.

The record separator is a new line and there is no field seperator.

How can I remove these invalid or special characters to creep in the records?
Please help me to find the logic in the shell sripting..

Thanks for the reply Perderabo. There are some more doubts;

=>There can be lots more invalid character like these, so shall I give all them in the 'like' command?

=>If I want to replace the 'invalid character' with a 'space' how can I do that?

=>How to run whole process for thousands of records in the file?

Attached is an example in this regard...

Attached Files

Example.txt (1.4 KB, 37 views)

The
echo "invalid characters like Å, å, Ä, ä or"
is providing the input data with illegal characters that need removal. I need some test data and this is one way to demo a command. And the command I am showing is
tr -dc " a-zA-Z0-9,\n"
and that is what removes the garbage. The tr command, in this form, lists the valid characters, not the invalid ones. You may need to add stuff to the list. To replace invalid characters with a space use I have switched to single quotes which may be better if you need certain special characters to be accepted. In your case you may want to just do Read the tr man page for more info.

I used the translate commands as follows:


1) tr -c ' a-zA-Z0-9,\n' ' ' < inputfile > outputfile ==> No result

2) tr -c '[:print:][:cntrl:]' ' ' < inputfile > outputfile ==>

The second command replaced one of the invalid character with space but retained all others. When I again ran the command on the resulted file, the complete files.(input as well as output.)

Please tell me some other combinations.

===================================================
Among the following characters, ~æ£ÇÄ, Ç was replaced.
===================================================

Thanks in advance.
Kanu

It is "Print". I dont know why this emoticon(invalid characters in my life?) came up.

Thanks

You can click "Disable smilies in text" option when you post. I edited your post to do that. Don't know why you're having trouble and I can't test in your environment but [:alnum:] is all letters and numbers. Try that. [:print:] is anything you can see, which is not what you want here. You want to discard some visible characters but keep others. Another thing to try is [a-z][A-Z][0-9] It's supposed to work without the brackets by some very old versions of tr required brackets for a range.