Sponsored Content
Top Forums Shell Programming and Scripting Check valid records in really big file with one commend.. Post 302196343 by vtischuk@yahoo. on Saturday 17th of May 2008 11:21:53 PM
Old 05-18-2008
Check valid records in really big file with one commend..

Hi,
I have a 5 gig file, no record terminators, field terminators are newline. The record length is 768 and I would like to check that every 768th byte is a newline and print out the byte position if it isn't. I would like to do this going either forward or backwards with one command if possible. I read that awk has a 3000 byte limit for a record so maybe a complex grep?

Thanks,

Victor
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Based on num of records in file1 need to check records in file2 to set some condns

Hi All, I have two files say file1 and file2. I want to check the number of records in file1 and if its atleast 2 (i.e., 2 or greater than 2 ) then I have to check records in file2 .If records in file2 is atleast 1 (i.e. if its not empty ) i have to set some conditions . Could you pls... (3 Replies)
Discussion started by: mavesum
3 Replies

2. Shell Programming and Scripting

how to check if a file has zero records

I want to do conditional logic on whether a file has 0 bytes or more than 0 bytes. what is the easiest way to do this? (2 Replies)
Discussion started by: guessingo
2 Replies

3. Shell Programming and Scripting

how to check for valid password

I need to check if an account has a valid password. Would something like this work? read ACCNAME if grep -q "^$ACCNAME:\$6:" /etc/shadow; thenI noticed every entry in my shadow file that has a password starts with $6 ... it works for my current setup, but would it always work? I can't test... (4 Replies)
Discussion started by: ADay2Long
4 Replies

4. Shell Programming and Scripting

Read a file with n records as one big string using linux

Hello! Is there a way i can read a file with n records as one big string using linux shell script? I have a file in the below format - REC1 REC2 REC3 . . . REC4 Record length is 3000 bytes per record and with a newline char at the end. What i need to do is - read this file as one... (5 Replies)
Discussion started by: mailme0205
5 Replies

5. Shell Programming and Scripting

Extract certain entries from big file:Request to check

Hi all I have a big file which I have attached here. And, I have to fetch certain entries and arrange in 5 columns Name Drug DAP ID disease approved or notIn the attached file data is arranged with tab separated columns in this way: and other data is... (2 Replies)
Discussion started by: manigrover
2 Replies

6. UNIX for Dummies Questions & Answers

Delete records from a big file based on some condition

Hi, To load a big file in a table,I have a make sure that all rows in the file has same number of the columns . So in my file if I am getting any rows which have columns not equal to 6 , I need to delete it . Delimiter is space and columns are optionally enclosed by "". This can be ... (1 Reply)
Discussion started by: hemantraijain
1 Replies

7. Shell Programming and Scripting

Read File and check records for length

I need a script that will run in unix to: 1) Read and input file with 1 column that contains for ex: 0123456789 1234567890 ...etc 2) Checks the first column if it is: a. Numeric from 0 - 9 b. if it is not less... (4 Replies)
Discussion started by: mrn6430
4 Replies

8. UNIX for Dummies Questions & Answers

How to check if file contains valid strings?

Hi All, I am a newbie...I would like to have a function which ll check if a file contains valid strings before "=" operator. Just to give you my requirement: assume my file has content: hello= gsdgsd sfdsg sgdsg sgdgdg world= gggg hhhh iiiii xxxx= pppp ppppp pppp my... (5 Replies)
Discussion started by: rtagarra
5 Replies

9. Shell Programming and Scripting

Check for valid hostnames

Hello, I am trying to develop a script to check for valid hostnames. Below are the prerequisites for a valid hostname which I got from wiki : Hostnames are composed of series of labels concatenated with dots, as are all domain names. For example, "en.wikipedia.org" is a hostname. Each label... (8 Replies)
Discussion started by: rahul2662
8 Replies

10. Shell Programming and Scripting

Script to check field value from a file records

I need a script to check the records in a file , if any value match transfer the record in error.txt file. 1- If any of the any field value is NULL(nothing in this field) Record1|Record2|Record3|Record4|Record5|DATE1|DATE2 Example: 11111111|22222222|NULL|12|444|27042018|27042018... (8 Replies)
Discussion started by: vivekn
8 Replies
SREC(5) 							GPSD Documentation							   SREC(5)

NAME
srec - Motorola S-record record and file format DESCRIPTION
Motorola S-records are a form of simple ASCII encoding for binary data. This format is commonly used for firmware uploads to GPSes, industrial robots, and other kinds of microcontroller-driven hardware. It has several convenient properties, including inspectability, easy editing with any text editor, and checksumming for verification of transmission across noisy serial lines. An S-record file consists of a sequence of specially formatted ASCII character strings. An S-record will be less than or equal to 78 bytes in length. The order of S-records within a file is of no significance and no particular order may be assumed. The general format of an S-record follows: +-------------------//------------------//-----------------------+ | type | count | address | data | checksum | +-------------------//------------------//-----------------------+ type A char[2] field. These characters describe the type of record (S0, S1, S2, S3, S5, S7, S8, or S9). count A char[2] field. These characters when paired and interpreted as a big-endian hexadecimal integer, display the count of remaining character pairs in the record. address A char[4,6, or 8] field. These characters grouped and interpreted as a big-endian hexadecimal integer, display the address at which the data field is to be loaded into memory. The length of the field depends on the number of bytes necessary to hold the address. A 2-byte address uses 4 characters, a 3-byte address uses 6 characters, and a 4-byte address uses 8 characters. data A char [0-64] field. These characters when paired and interpreted as hexadecimal values represent the memory loadable data or descriptive information. checksum A char[2] field. These characters when paired and interpreted as a big-endian hexadecimal integer display the least significant byte of the ones complement of the sum of the byte values represented by the pairs of characters making up the count, the address, and the data fields. Each record is terminated with a line feed. If any additional or different record terminator(s) or delay characters are needed during transmission to the target system it is the responsibility of the transmitting program to provide them. There are 9 record types, as follows: S0 The type of record is 'S0' (0x5330). The address field is unused and will be filled with zeros (0x0000). The header information within the data field is divided into the following subfields. 1. mname is char[20] and is the module name. 2. ver is char[2] and is the version number. 3. rev is char[2] and is the revision number. 4. description is char[0-36] and is a text comment. Each of the subfields is composed of ASCII bytes whose associated characters, when paired, represent one byte hexadecimal values in the case of the version and revision numbers, or represent the hexadecimal values of the ASCII characters comprising the module name and description. S1 The type of record field is 'S1' (0x5331). The address field is interpreted as a 2-byte big-endian address. The data field is composed of memory loadable data. S2 The type of record field is 'S2' (0x5332). The address field is interpreted as a 3-byte big-endian address. The data field is composed of memory loadable data. S3 The type of record field is 'S3' (0x5333). The address field is interpreted as a 4-byte big-endian address. The data field is composed of memory loadable data. S5 The type of record field is 'S5' (0x5335). The address field is interpreted as a 2-byte big-endian value and contains the count of S1, S2, and S3 records previously transmitted. There is no data field. S7 The type of record field is 'S7' (0x5337). The address field contains the starting execution address and is interpreted as a 4-byte big-endian address. There is no data field. S8 The type of record field is 'S8' (0x5338). The address field contains the starting execution address and is interpreted as a 3-byte big-endian address. There is no data field. S9 The type of record field is 'S9' (0x5339). The address field contains the starting execution address and is interpreted as a 2-byte big-endian address. There is no data field. EXAMPLE
Shown below is a typical S-record format file. S00600004844521B S1130000285F245F2212226A000424290008237C2A S11300100002000800082629001853812341001813 S113002041E900084E42234300182342000824A952 S107003000144ED492 S5030004F8 S9030000FC The file consists of one S0 record, four S1 records, one S5 record and an S9 record. The S0 record is comprised as follows: o S0 S-record type S0, indicating it is a header record. o 06 Hexadecimal 06 (decimal 6), indicating that six character pairs (or ASCII bytes) follow. o 00 00 Four character 2-byte address field, zeroes in this example. o 48 44 52 ASCII H, D, and R - "HDR". o 1B The checksum. The first S1 record is comprised as follows: o S1 S-record type S1, indicating it is a data record to be loaded at a 2-byte address. o 13 Hexadecimal 13 (decimal 19), indicating that nineteen character pairs, representing a 2 byte address, 16 bytes of binary data, and a 1 byte checksum, follow. o 00 00 Four character 2-byte address field; hexidecimal address 0x0000, where the data which follows is to be loaded. o 28 5F 24 5F 22 12 22 6A 00 04 24 29 00 08 23 7C Sixteen character pairs representing the actual binary data. o 2A The checksum. The second and third S1 records each contain 0x13 (19) character pairs and are ended with checksums of 13 and 52, respectively. The fourth S1 record contains 07 character pairs and has a checksum of 92. The S5 record is comprised as follows: o S5 S-record type S5, indicating it is a count record indicating the number of S1 records o 03 Hexadecimal 03 (decimal 3), indicating that three character pairs follow. o 00 04 Hexadecimal 0004 (decimal 4), indicating that there are four data records previous to this record. o F8 The checksum. The S9 record is comprised as follows: o S9 S-record type S9, indicating it is a termination record. o 03 Hexadecimal 03 (decimal 3), indicating that three character pairs follow. o 00 00 The address field, hexadecimal 0 (decimal 0) indicating the starting execution address. o FC The checksum. NOTES
o There isn't any evidence that Motorola ever made use of the header information within the data field of the S0 record, as described above. This may have been used by some third party vendors. o The Unix manual page on S-records is the only place that a 78-byte limit on total record length or 64-byte limit on data length is documented. These values shouldn't be trusted for the general case. o The count field can have values in the range of 0x3 (2 bytes of address + 1 byte checksum = 3, a not very useful record) to 0xff; this is the count of remaining character pairs, including checksum. o If you write code to convert S-Records, you should always assume that a record can be as long as 514 (decimal) characters in length (255 * 2 = 510, plus 4 characters for the type and count fields), plus any terminating character(s). That is, in establishing an input buffer in C, you would declare it to be an array of 515 chars, thus leaving room for the terminating null character. SEE ALSO
gpsd(8), gps(1), libgps(3), libgpsd(3), gpsfake(1). gpsprof(1). AUTHOR
From an anonymous web page, itself claiming to have been derived from an old Unix manual page. Now maintained by the GPSD project, which added endianness clarifications. The GPSD Project 15 Jul 2005 SREC(5)
All times are GMT -4. The time now is 06:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy