Sponsored Content
Top Forums UNIX for Dummies Questions & Answers To get the invalid characters from a file Post 302747347 by schandru on Friday 21st of December 2012 06:04:20 AM
Old 12-21-2012
Hi,
Today i faced a problem in which the file at Unix had the characters like � and it caused the file processing failure at downstream end which is Mainframe. and it somehow identified by the Mainframe and notified to me, so i removed the above mentioned by going to the particular line and removed the characters and it got processed successfully.

Could you please help me to find these characters like non-standard characters in a file?

Hope now this makes sense and clear..!

Chandru.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Invalid Characters in the file.

I am working on AIX. We ftp files to a database. The flat files are having thousands of records and each record is having some 50 to 60 characters(there are fields having certain character length). In addition to some valid ascii characters some invalid characters like Å, å, Ä, ä or pipes creep in which... (5 Replies)
Discussion started by: kanu_pathak
5 Replies

2. UNIX for Dummies Questions & Answers

Invalid Characters in the file.

I am working on AIX. We ftp files to a database. The flat files are having thousands of records and each record is having some 50 to 60 characters(there are fields having certain character length). In addition to some valid ascii characters some invalid characters like Å, å, Ä, ä or pipes creep in which... (15 Replies)
Discussion started by: kanu_pathak
15 Replies

3. Programming

string with invalid characters

This is a pretty straight-forward question. Within a program of mine, I have a string that's going to be used as a filename, but it might have some invalid characters in it that wouldn't be valid in a filename. If there are any invalid characters, I want to get rid of them and essentially squeeze... (4 Replies)
Discussion started by: cleopard
4 Replies

4. Shell Programming and Scripting

writing shell script to find line of invalid characters

Hi, I have to write s script to check an input file for invalid characters. In this script I have to find the exact line of the invalid character. If the input file contain 2 invalid character sat line 10 and 17, the script will show the value 10 and 17. Any help is appreciated. (3 Replies)
Discussion started by: beginner82
3 Replies

5. UNIX for Dummies Questions & Answers

to delete an invalid file

there is a file is generated from my program due to undefined filename. -rw-r--r-- 1 angie angie 8644055 Jun 22 09:17 Ô$ÿÿÿÿÿÆ may i know how to delete this file..??? thanks in advance... :) (5 Replies)
Discussion started by: chxxangie
5 Replies

6. Shell Programming and Scripting

Capturing the invalid records to error file

HI, I have a source file which has the below data. Tableid,table.txt sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6 Tableid,table sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6 Tableid,table.txt sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6 Tableid,table sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6... (6 Replies)
Discussion started by: shruthidwh
6 Replies

7. Shell Programming and Scripting

Remove invalid database characters on a file

Hi All - I'm building a script wherein it is design to remove characters that are not accepted on a non-unicode database. Examples are the following: ï,¿,½,Â,é, etc. I can easily sed those characters one-by-one but I there's a problem when other unicode characters are found. Is there any way to... (1 Reply)
Discussion started by: Jin_
1 Replies

8. Shell Programming and Scripting

Valid and invalid date in the file

Hi All, How to validate the 4th column,it is date column in the file, if it valid move to valid file else moved invalid file. 9f680174-cb87|20077337254|0|20120511|N 9f680174-cb88|20077337254|0|20120534|N i want two file valid.txt and invalid.txt Thanks, (7 Replies)
Discussion started by: bmk
7 Replies

9. Shell Programming and Scripting

How to get the Invalid records from a file using awk?

My Input file is fixed length record ends with . as end of the line and the character length is 4156 Example: 12234XYZ TY^4253$+00000-00000........... I need to check is there any control characters(like ^M,^Z) The line will be splitted awk '{id=substr($0,1,5) nm=substr($0,6,3)... (2 Replies)
Discussion started by: dineshaila
2 Replies

10. UNIX for Beginners Questions & Answers

Search for the invalid url in a file

Hello guys, Here i am writing a script to check for a valid url from a file,i am getting the valid url & i print it in a file and i want to print the invalid url also.how to do that? #here is my script if then URL=$(grep -E -o... (2 Replies)
Discussion started by: Meeran Rizvi
2 Replies
utf8trans(1)							     docbook2X							      utf8trans(1)

NAME
utf8trans - Transliterate UTF-8 characters according to a table SYNOPSIS
utf8trans charmap [file]... DESCRIPTION
utf8trans transliterates characters in the specified files (or standard input, if they are not specified) and writes the output to standard output. All input and output is in the UTF-8 encoding. This program is usually used to render characters in Unicode text files as some markup escapes or ASCII transliterations. (It is not in- tended for general charset conversions.) It provides functionality similar to the character maps in XSLT 2.0 (XML Stylesheet Language - Transformations, version 2.0). OPTIONS
-m, --modify Modifies the given files in-place with their transliterated output, instead of sending it to standard output. This option is useful for efficient transliteration of many files at once. --help Show brief usage information and exit. --version Show version and exit. USAGE
The translation is done according to the rules in the 'character map', named in the file charmap. It has the following format: 1. Each line represents a translation entry, except for blank lines and comment lines, which are ignored. 2. Any amount of whitespace (space or tab) may precede the start of an entry. 3. Comment lines begin with #. Everything on the same line is ignored. 4. Each entry consists of the Unicode codepoint of the character to translate, in hexadecimal, followed one space or tab, followed by the translation string, up to the end of the line. 5. The translation string is taken literally, including any leading and trailing spaces (except the delimeter between the codepoint and the translation string), and all types of characters. The newline at the end is not included. The above format is intended to be restrictive, to keep utf8trans simple. But if a XML-based format is desired, there is a xmlcharmap2utf8trans script that comes with the docbook2X distribution, that converts character maps in XSLT 2.0 format to the utf8trans format. LIMITATIONS
o utf8trans does not work with binary files, because malformed UTF-8 sequences in the input are substituted with U+FFFD characters. Howev- er, null characters in the input are handled correctly. This limitation may be removed in the future. o There is no way to include a newline or null in the substitution string. AUTHOR
Steve Cheng <stevecheng@users.sourceforge.net>. docbook2X 0.8.8 3 March 2007 utf8trans(1)
All times are GMT -4. The time now is 08:27 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy