Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Invalid Characters in the file. Post 302163183 by Perderabo on Thursday 31st of January 2008 07:21:31 AM
Old 01-31-2008
The
echo "invalid characters like Å, å, Ä, ä or"
is providing the input data with illegal characters that need removal. I need some test data and this is one way to demo a command. And the command I am showing is
tr -dc " a-zA-Z0-9,\n"
and that is what removes the garbage. The tr command, in this form, lists the valid characters, not the invalid ones. You may need to add stuff to the list. To replace invalid characters with a space use
Code:
$ echo "invalid characters like Å, å, Ä, ä or"| tr -c ' a-zA-Z0-9,\n'  ' '
invalid characters like  ,  ,  ,   or
$

I have switched to single quotes which may be better if you need certain special characters to be accepted. In your case you may want to just do
Code:
tr -c ' a-zA-Z0-9,\n'  ' ' < inputfile > outputfile

Read the tr man page for more info.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Invalid Characters in the file.

I am working on AIX. We ftp files to a database. The flat files are having thousands of records and each record is having some 50 to 60 characters(there are fields having certain character length). In addition to some valid ascii characters some invalid characters like Å, å, Ä, ä or pipes creep in which... (5 Replies)
Discussion started by: kanu_pathak
5 Replies

2. Programming

string with invalid characters

This is a pretty straight-forward question. Within a program of mine, I have a string that's going to be used as a filename, but it might have some invalid characters in it that wouldn't be valid in a filename. If there are any invalid characters, I want to get rid of them and essentially squeeze... (4 Replies)
Discussion started by: cleopard
4 Replies

3. Shell Programming and Scripting

writing shell script to find line of invalid characters

Hi, I have to write s script to check an input file for invalid characters. In this script I have to find the exact line of the invalid character. If the input file contain 2 invalid character sat line 10 and 17, the script will show the value 10 and 17. Any help is appreciated. (3 Replies)
Discussion started by: beginner82
3 Replies

4. UNIX for Dummies Questions & Answers

to delete an invalid file

there is a file is generated from my program due to undefined filename. -rw-r--r-- 1 angie angie 8644055 Jun 22 09:17 Ô$ÿÿÿÿÿÆ may i know how to delete this file..??? thanks in advance... :) (5 Replies)
Discussion started by: chxxangie
5 Replies

5. Shell Programming and Scripting

Capturing the invalid records to error file

HI, I have a source file which has the below data. Tableid,table.txt sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6 Tableid,table sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6 Tableid,table.txt sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6 Tableid,table sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6... (6 Replies)
Discussion started by: shruthidwh
6 Replies

6. Shell Programming and Scripting

Remove invalid database characters on a file

Hi All - I'm building a script wherein it is design to remove characters that are not accepted on a non-unicode database. Examples are the following: ï,¿,½,Â,é, etc. I can easily sed those characters one-by-one but I there's a problem when other unicode characters are found. Is there any way to... (1 Reply)
Discussion started by: Jin_
1 Replies

7. Shell Programming and Scripting

Valid and invalid date in the file

Hi All, How to validate the 4th column,it is date column in the file, if it valid move to valid file else moved invalid file. 9f680174-cb87|20077337254|0|20120511|N 9f680174-cb88|20077337254|0|20120534|N i want two file valid.txt and invalid.txt Thanks, (7 Replies)
Discussion started by: bmk
7 Replies

8. UNIX for Dummies Questions & Answers

To get the invalid characters from a file

Hello, Can any one help me in below query to search all the invalid characters that UNIX cannot recognize from a file. can we do anything with the help of grep command or any other commands. Also, i am not sure what are the invalid characters present in the file. Many thanks in advance. ... (6 Replies)
Discussion started by: schandru
6 Replies

9. Shell Programming and Scripting

How to get the Invalid records from a file using awk?

My Input file is fixed length record ends with . as end of the line and the character length is 4156 Example: 12234XYZ TY^4253$+00000-00000........... I need to check is there any control characters(like ^M,^Z) The line will be splitted awk '{id=substr($0,1,5) nm=substr($0,6,3)... (2 Replies)
Discussion started by: dineshaila
2 Replies

10. UNIX for Beginners Questions & Answers

Search for the invalid url in a file

Hello guys, Here i am writing a script to check for a valid url from a file,i am getting the valid url & i print it in a file and i want to print the invalid url also.how to do that? #here is my script if then URL=$(grep -E -o... (2 Replies)
Discussion started by: Meeran Rizvi
2 Replies
ICONV(1)						     Linux Programmer's Manual							  ICONV(1)

NAME
iconv - character set conversion SYNOPSIS
iconv [OPTION...] [-f encoding] [-t encoding] [inputfile ...] iconv -l DESCRIPTION
The iconv program converts text from one encoding to another encoding. More precisely, it converts from the encoding given for the -f option to the encoding given for the -t option. Either of these encodings defaults to the encoding of the current locale. All the input- files are read and converted in turn; if no inputfile is given, the standard input is used. The converted text is printed to standard out- put. The encodings permitted are system dependent. For the libiconv implementation, they are listed in the iconv_open(3) manual page. Options controlling the input and output format: -f encoding, --from-code=encoding Specifies the encoding of the input. -t encoding, --to-code=encoding Specifies the encoding of the output. Options controlling conversion problems: -c When this option is given, characters that cannot be converted are silently discarded, instead of leading to a conversion error. --unicode-subst=formatstring When this option is given, Unicode characters that cannot be represented in the target encoding are replaced with a placeholder string that is constructed from the given formatstring, applied to the Unicode code point. The formatstring must be a format string in the same format as for the printf command or the printf() function, taking either no argument or exactly one unsigned integer argument. --byte-subst=formatstring When this option is given, bytes in the input that are not valid in the source encoding are replaced with a placeholder string that is constructed from the given formatstring, applied to the byte's value. The formatstring must be a format string in the same format as for the printf command or the printf() function, taking either no argument or exactly one unsigned integer argument. --widechar-subst=formatstring When this option is given, wide characters in the input that are not valid in the source encoding are replaced with a placeholder string that is constructed from the given formatstring, applied to the byte's value. The formatstring must be a format string in the same format as for the printf command or the printf() function, taking either no argument or exactly one unsigned integer argument. Options controlling error output: -s, --silent When this option is given, error messages about invalid or unconvertible characters are omitted, but the actual converted text is unaffected. The iconv -l or iconv --list command lists the names of the supported encodings, in a system dependent format. For the libiconv implementa- tion, the names are printed in upper case, separated by whitespace, and alias names of an encoding are listed on the same line as the encoding itself. EXAMPLES
iconv -f ISO-8859-1 -t UTF-8 converts input from the old West-European encoding ISO-8859-1 to Unicode. iconv -f KOI8-R --byte-subst="<0x%x>" --unicode-subst="<U+%04X>" converts input from the old Russian encoding KOI8-R to the locale encoding, substituting an angle bracket notation with hexadecimal numbers for invalid bytes and for valid but unconvertible characters. iconv --list lists the supported encodings. SEE ALSO
iconv_open(3) GNU
January 22, 2006 ICONV(1)
All times are GMT -4. The time now is 04:37 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy