Sponsored Content
Top Forums Shell Programming and Scripting Invalid Characters in the file. Post 302163112 by kanu_pathak on Thursday 31st of January 2008 04:24:33 AM
Old 01-31-2008
Invalid Characters in the file.

I am working on AIX. We ftp files to a database. The flat files are having thousands of records and each record is having some 50 to 60 characters(there are fields having certain character length). In addition to some valid ascii characters some invalid characters like Å, å, Ä, ä or pipes creep in which datawarehouse rejects to load in.
Example: AcuM-^?a 051706 ;
above is a field in the record which is having special characters like -,^ and ? , which should not have been there.

The record separator is a new line and there is no field seperator.

How can I remove these invalid or special characters to creep in the records?
Please help me to find the logic in the shell sripting..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Invalid Characters in the file.

I am working on AIX. We ftp files to a database. The flat files are having thousands of records and each record is having some 50 to 60 characters(there are fields having certain character length). In addition to some valid ascii characters some invalid characters like Å, å, Ä, ä or pipes creep in which... (15 Replies)
Discussion started by: kanu_pathak
15 Replies

2. Programming

string with invalid characters

This is a pretty straight-forward question. Within a program of mine, I have a string that's going to be used as a filename, but it might have some invalid characters in it that wouldn't be valid in a filename. If there are any invalid characters, I want to get rid of them and essentially squeeze... (4 Replies)
Discussion started by: cleopard
4 Replies

3. Shell Programming and Scripting

writing shell script to find line of invalid characters

Hi, I have to write s script to check an input file for invalid characters. In this script I have to find the exact line of the invalid character. If the input file contain 2 invalid character sat line 10 and 17, the script will show the value 10 and 17. Any help is appreciated. (3 Replies)
Discussion started by: beginner82
3 Replies

4. UNIX for Dummies Questions & Answers

to delete an invalid file

there is a file is generated from my program due to undefined filename. -rw-r--r-- 1 angie angie 8644055 Jun 22 09:17 Ô$ÿÿÿÿÿÆ may i know how to delete this file..??? thanks in advance... :) (5 Replies)
Discussion started by: chxxangie
5 Replies

5. Shell Programming and Scripting

Capturing the invalid records to error file

HI, I have a source file which has the below data. Tableid,table.txt sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6 Tableid,table sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6 Tableid,table.txt sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6 Tableid,table sourceid,1,2,3,4,5,6 targetid,1,2,3,4,5,6... (6 Replies)
Discussion started by: shruthidwh
6 Replies

6. Shell Programming and Scripting

Remove invalid database characters on a file

Hi All - I'm building a script wherein it is design to remove characters that are not accepted on a non-unicode database. Examples are the following: ï,¿,½,Â,é, etc. I can easily sed those characters one-by-one but I there's a problem when other unicode characters are found. Is there any way to... (1 Reply)
Discussion started by: Jin_
1 Replies

7. Shell Programming and Scripting

Valid and invalid date in the file

Hi All, How to validate the 4th column,it is date column in the file, if it valid move to valid file else moved invalid file. 9f680174-cb87|20077337254|0|20120511|N 9f680174-cb88|20077337254|0|20120534|N i want two file valid.txt and invalid.txt Thanks, (7 Replies)
Discussion started by: bmk
7 Replies

8. UNIX for Dummies Questions & Answers

To get the invalid characters from a file

Hello, Can any one help me in below query to search all the invalid characters that UNIX cannot recognize from a file. can we do anything with the help of grep command or any other commands. Also, i am not sure what are the invalid characters present in the file. Many thanks in advance. ... (6 Replies)
Discussion started by: schandru
6 Replies

9. Shell Programming and Scripting

How to get the Invalid records from a file using awk?

My Input file is fixed length record ends with . as end of the line and the character length is 4156 Example: 12234XYZ TY^4253$+00000-00000........... I need to check is there any control characters(like ^M,^Z) The line will be splitted awk '{id=substr($0,1,5) nm=substr($0,6,3)... (2 Replies)
Discussion started by: dineshaila
2 Replies

10. UNIX for Beginners Questions & Answers

Search for the invalid url in a file

Hello guys, Here i am writing a script to check for a valid url from a file,i am getting the valid url & i print it in a file and i want to print the invalid url also.how to do that? #here is my script if then URL=$(grep -E -o... (2 Replies)
Discussion started by: Meeran Rizvi
2 Replies
SREC(5) 							GPSD Documentation							   SREC(5)

NAME
srec - Motorola S-record record and file format DESCRIPTION
Motorola S-records are a form of simple ASCII encoding for binary data. This format is commonly used for firmware uploads to GPSes, industrial robots, and other kinds of microcontroller-driven hardware. It has several convenient properties, including inspectability, easy editing with any text editor, and checksumming for verification of transmission across noisy serial lines. An S-record file consists of a sequence of specially formatted ASCII character strings. An S-record will be less than or equal to 78 bytes in length. The order of S-records within a file is of no significance and no particular order may be assumed. The general format of an S-record follows: +-------------------//------------------//-----------------------+ | type | count | address | data | checksum | +-------------------//------------------//-----------------------+ type A char[2] field. These characters describe the type of record (S0, S1, S2, S3, S5, S7, S8, or S9). count A char[2] field. These characters when paired and interpreted as a big-endian hexadecimal integer, display the count of remaining character pairs in the record. address A char[4,6, or 8] field. These characters grouped and interpreted as a big-endian hexadecimal integer, display the address at which the data field is to be loaded into memory. The length of the field depends on the number of bytes necessary to hold the address. A 2-byte address uses 4 characters, a 3-byte address uses 6 characters, and a 4-byte address uses 8 characters. data A char [0-64] field. These characters when paired and interpreted as hexadecimal values represent the memory loadable data or descriptive information. checksum A char[2] field. These characters when paired and interpreted as a big-endian hexadecimal integer display the least significant byte of the ones complement of the sum of the byte values represented by the pairs of characters making up the count, the address, and the data fields. Each record is terminated with a line feed. If any additional or different record terminator(s) or delay characters are needed during transmission to the target system it is the responsibility of the transmitting program to provide them. There are 9 record types, as follows: S0 The type of record is 'S0' (0x5330). The address field is unused and will be filled with zeros (0x0000). The header information within the data field is divided into the following subfields. 1. mname is char[20] and is the module name. 2. ver is char[2] and is the version number. 3. rev is char[2] and is the revision number. 4. description is char[0-36] and is a text comment. Each of the subfields is composed of ASCII bytes whose associated characters, when paired, represent one byte hexadecimal values in the case of the version and revision numbers, or represent the hexadecimal values of the ASCII characters comprising the module name and description. S1 The type of record field is 'S1' (0x5331). The address field is interpreted as a 2-byte big-endian address. The data field is composed of memory loadable data. S2 The type of record field is 'S2' (0x5332). The address field is interpreted as a 3-byte big-endian address. The data field is composed of memory loadable data. S3 The type of record field is 'S3' (0x5333). The address field is interpreted as a 4-byte big-endian address. The data field is composed of memory loadable data. S5 The type of record field is 'S5' (0x5335). The address field is interpreted as a 2-byte big-endian value and contains the count of S1, S2, and S3 records previously transmitted. There is no data field. S7 The type of record field is 'S7' (0x5337). The address field contains the starting execution address and is interpreted as a 4-byte big-endian address. There is no data field. S8 The type of record field is 'S8' (0x5338). The address field contains the starting execution address and is interpreted as a 3-byte big-endian address. There is no data field. S9 The type of record field is 'S9' (0x5339). The address field contains the starting execution address and is interpreted as a 2-byte big-endian address. There is no data field. EXAMPLE
Shown below is a typical S-record format file. S00600004844521B S1130000285F245F2212226A000424290008237C2A S11300100002000800082629001853812341001813 S113002041E900084E42234300182342000824A952 S107003000144ED492 S5030004F8 S9030000FC The file consists of one S0 record, four S1 records, one S5 record and an S9 record. The S0 record is comprised as follows: o S0 S-record type S0, indicating it is a header record. o 06 Hexadecimal 06 (decimal 6), indicating that six character pairs (or ASCII bytes) follow. o 00 00 Four character 2-byte address field, zeroes in this example. o 48 44 52 ASCII H, D, and R - "HDR". o 1B The checksum. The first S1 record is comprised as follows: o S1 S-record type S1, indicating it is a data record to be loaded at a 2-byte address. o 13 Hexadecimal 13 (decimal 19), indicating that nineteen character pairs, representing a 2 byte address, 16 bytes of binary data, and a 1 byte checksum, follow. o 00 00 Four character 2-byte address field; hexidecimal address 0x0000, where the data which follows is to be loaded. o 28 5F 24 5F 22 12 22 6A 00 04 24 29 00 08 23 7C Sixteen character pairs representing the actual binary data. o 2A The checksum. The second and third S1 records each contain 0x13 (19) character pairs and are ended with checksums of 13 and 52, respectively. The fourth S1 record contains 07 character pairs and has a checksum of 92. The S5 record is comprised as follows: o S5 S-record type S5, indicating it is a count record indicating the number of S1 records o 03 Hexadecimal 03 (decimal 3), indicating that three character pairs follow. o 00 04 Hexadecimal 0004 (decimal 4), indicating that there are four data records previous to this record. o F8 The checksum. The S9 record is comprised as follows: o S9 S-record type S9, indicating it is a termination record. o 03 Hexadecimal 03 (decimal 3), indicating that three character pairs follow. o 00 00 The address field, hexadecimal 0 (decimal 0) indicating the starting execution address. o FC The checksum. NOTES
o There isn't any evidence that Motorola ever made use of the header information within the data field of the S0 record, as described above. This may have been used by some third party vendors. o The Unix manual page on S-records is the only place that a 78-byte limit on total record length or 64-byte limit on data length is documented. These values shouldn't be trusted for the general case. o The count field can have values in the range of 0x3 (2 bytes of address + 1 byte checksum = 3, a not very useful record) to 0xff; this is the count of remaining character pairs, including checksum. o If you write code to convert S-Records, you should always assume that a record can be as long as 514 (decimal) characters in length (255 * 2 = 510, plus 4 characters for the type and count fields), plus any terminating character(s). That is, in establishing an input buffer in C, you would declare it to be an array of 515 chars, thus leaving room for the terminating null character. SEE ALSO
gpsd(8), gps(1), libgps(3), libgpsd(3), gpsfake(1). gpsprof(1). AUTHOR
From an anonymous web page, itself claiming to have been derived from an old Unix manual page. Now maintained by the GPSD project, which added endianness clarifications. The GPSD Project 15 Jul 2005 SREC(5)
All times are GMT -4. The time now is 03:52 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy