Sponsored Content
Top Forums Shell Programming and Scripting Linux shell script to insert new lines based on delimiter count Post 302986016 by digitalnirvana on Friday 18th of November 2016 09:09:11 AM
Old 11-18-2016
Apologies, my bad. I should've uploaded the file. Attached is a masked .dat file renamed as .txt for uploading.

On opening it with notepad++ in Windows, the null characters show up as boxes. In a Linux vi the nulls are ^@.

All the records in this file are in one row. This particular file has 2 records followed by the trailer record.
  • First record = starts at the beginning of the file 00000230 (this field gives the length of the record in bytes)
  • Second record = starts at the next 00000230 (it is a coincidence, here both records have same length)
  • Trailer record = starts at 0000096 (the trailer length is of 96 bytes and it also has 80 delimiters of ^@ or null characters. Ignore my earlier post saying trailer has 20 delimiters. It has 80 actually)

As the field lengths are variable so we cannot define a record in terms of total length of its fields or total bytes. This is why we are defining a record as effectively having length of 80 ^@ delimiters.

I require the 1st record in one row, 2nd record in next row and so on till the end of the file, with the trailer in the last row. If there is a way of adding a newline after every 80th ^@ from the beginning till the eof, then perhaps it will work?

The only unprintable character is null ^@, no TABS or other spaces, all other characters are alphanumeric.

Please let me know if any questions. Thanks for the help
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Perl/shell script count the lines

Hi Guys, I want to write a perl/shell script do parse the following file input file content NPA-NXX SC 2084549 45 2084552 45 2084563 2007 2084572 45 2084580 45 3278411 45 3278430 45 3278493 530 3278507 530... (3 Replies)
Discussion started by: pistachio
3 Replies

2. Shell Programming and Scripting

insert leading zeroes based on the character count

Hi, I need add leading zeroes to a field in a file based on the character count. The field can be of 1 character to 6 character length. I need to make the field 14bytes. eg: 8351,20,1 8351,234,6 8351,2,0 8351,1234,2 8351,123456,1 8351,12345,2 This should become. ... (3 Replies)
Discussion started by: gpaulose
3 Replies

3. Shell Programming and Scripting

Shell script to put delimiter for a no delimiter variable length text file

Hi, I have a No Delimiter variable length text file with following schema - Column Name Data length Firstname 5 Lastname 5 age 3 phoneno1 10 phoneno2 10 phoneno3 10 sample data - ... (16 Replies)
Discussion started by: Gaurav Martha
16 Replies

4. Shell Programming and Scripting

Bash script to count and insert

Hi not sure if this is possible but I need some help with a bash script, I have a text file and on the first line that starts with 7150230 I need it to put a 1 at position 79 and a 2 at position 88, this is where it gets complicated, on the next line it finds that starts with 7150230 I then need it... (8 Replies)
Discussion started by: firefox2k2
8 Replies

5. Shell Programming and Scripting

Insert Columns before the last Column based on the Count of Delimiters

Hi, I have a requirement where in I need to insert delimiters before the last column of the total delimiters is less than a specified number. Say if the delimiters is less than 139, I need to insert 2 columns ( with blanks) before the last field awk -F 'Ç' '{ if (NF-1 < 139)} END { "Insert 2... (5 Replies)
Discussion started by: arunkesi
5 Replies

6. Shell Programming and Scripting

File Count Based on FileDate using Shell Script

I have file listed in my directory in following format -rwxrwxr-x+ 1 test test 4.9M Oct 3 16:06 test20141002150108.txt -rwxrwxr-x+ 1 test test 4.9M Oct 4 16:06 test20141003150108.txt -rwxrwxr-x+ 1 test test 4.9M Oct 5 16:06 test20141005150108.txt -rwxrwxr-x+ 1 test ... (2 Replies)
Discussion started by: krish2014
2 Replies

7. Shell Programming and Scripting

Removing duplicate lines on first column based with pipe delimiter

Hi, I have tried to remove dublicate lines based on first column with pipe delimiter . but i ma not able to get some uniqu lines Command : sort -t'|' -nuk1 file.txt Input : 38376KZ|09/25/15|1.057 38376KZ|09/25/15|1.057 02006YB|09/25/15|0.859 12593PS|09/25/15|2.803... (2 Replies)
Discussion started by: parithi06
2 Replies

8. Shell Programming and Scripting

awk joining multiple lines based on field count

Hi Folks, I have a file with fields as follows which has last field in multiple lines. I would like to combine a line which has three fields with single field line for as shown in expected output. Please help. INPUT hname01 windows appnamec1eda_p1, ... (5 Replies)
Discussion started by: shunya
5 Replies

9. Shell Programming and Scripting

Shell script count lines and sum numbers from multiple files

I want to count the number of lines, I need this result be a number, and sum the last numeric column, I had done to make this one at time, but I need to make this for a crontab, so, it has to be an script, here is my lines: It counts the number of lines: egrep -i String file_name_201611* |... (5 Replies)
Discussion started by: Elly
5 Replies

10. Shell Programming and Scripting

Split files based on row delimiter count

I have a huge file (around 4-5 GB containing 20 million rows) which has text like: <EOFD>11<EOFD>22<EORD>2<EOFD>2222<EOFD>3333<EORD>3<EOFD>44<EOFD>55<EORD>66<EOFD>888<EOFD>9999<EORD> Actually above is an extracted file from a Sql Server with each field delimited by <EOFD> and each row ends... (8 Replies)
Discussion started by: amvip
8 Replies
SREC(5) 							GPSD Documentation							   SREC(5)

NAME
srec - Motorola S-record record and file format DESCRIPTION
Motorola S-records are a form of simple ASCII encoding for binary data. This format is commonly used for firmware uploads to GPSes, industrial robots, and other kinds of microcontroller-driven hardware. It has several convenient properties, including inspectability, easy editing with any text editor, and checksumming for verification of transmission across noisy serial lines. An S-record file consists of a sequence of specially formatted ASCII character strings. An S-record will be less than or equal to 78 bytes in length. The order of S-records within a file is of no significance and no particular order may be assumed. The general format of an S-record follows: +-------------------//------------------//-----------------------+ | type | count | address | data | checksum | +-------------------//------------------//-----------------------+ type A char[2] field. These characters describe the type of record (S0, S1, S2, S3, S5, S7, S8, or S9). count A char[2] field. These characters when paired and interpreted as a big-endian hexadecimal integer, display the count of remaining character pairs in the record. address A char[4,6, or 8] field. These characters grouped and interpreted as a big-endian hexadecimal integer, display the address at which the data field is to be loaded into memory. The length of the field depends on the number of bytes necessary to hold the address. A 2-byte address uses 4 characters, a 3-byte address uses 6 characters, and a 4-byte address uses 8 characters. data A char [0-64] field. These characters when paired and interpreted as hexadecimal values represent the memory loadable data or descriptive information. checksum A char[2] field. These characters when paired and interpreted as a big-endian hexadecimal integer display the least significant byte of the ones complement of the sum of the byte values represented by the pairs of characters making up the count, the address, and the data fields. Each record is terminated with a line feed. If any additional or different record terminator(s) or delay characters are needed during transmission to the target system it is the responsibility of the transmitting program to provide them. There are 9 record types, as follows: S0 The type of record is 'S0' (0x5330). The address field is unused and will be filled with zeros (0x0000). The header information within the data field is divided into the following subfields. 1. mname is char[20] and is the module name. 2. ver is char[2] and is the version number. 3. rev is char[2] and is the revision number. 4. description is char[0-36] and is a text comment. Each of the subfields is composed of ASCII bytes whose associated characters, when paired, represent one byte hexadecimal values in the case of the version and revision numbers, or represent the hexadecimal values of the ASCII characters comprising the module name and description. S1 The type of record field is 'S1' (0x5331). The address field is interpreted as a 2-byte big-endian address. The data field is composed of memory loadable data. S2 The type of record field is 'S2' (0x5332). The address field is interpreted as a 3-byte big-endian address. The data field is composed of memory loadable data. S3 The type of record field is 'S3' (0x5333). The address field is interpreted as a 4-byte big-endian address. The data field is composed of memory loadable data. S5 The type of record field is 'S5' (0x5335). The address field is interpreted as a 2-byte big-endian value and contains the count of S1, S2, and S3 records previously transmitted. There is no data field. S7 The type of record field is 'S7' (0x5337). The address field contains the starting execution address and is interpreted as a 4-byte big-endian address. There is no data field. S8 The type of record field is 'S8' (0x5338). The address field contains the starting execution address and is interpreted as a 3-byte big-endian address. There is no data field. S9 The type of record field is 'S9' (0x5339). The address field contains the starting execution address and is interpreted as a 2-byte big-endian address. There is no data field. EXAMPLE
Shown below is a typical S-record format file. S00600004844521B S1130000285F245F2212226A000424290008237C2A S11300100002000800082629001853812341001813 S113002041E900084E42234300182342000824A952 S107003000144ED492 S5030004F8 S9030000FC The file consists of one S0 record, four S1 records, one S5 record and an S9 record. The S0 record is comprised as follows: o S0 S-record type S0, indicating it is a header record. o 06 Hexadecimal 06 (decimal 6), indicating that six character pairs (or ASCII bytes) follow. o 00 00 Four character 2-byte address field, zeroes in this example. o 48 44 52 ASCII H, D, and R - "HDR". o 1B The checksum. The first S1 record is comprised as follows: o S1 S-record type S1, indicating it is a data record to be loaded at a 2-byte address. o 13 Hexadecimal 13 (decimal 19), indicating that nineteen character pairs, representing a 2 byte address, 16 bytes of binary data, and a 1 byte checksum, follow. o 00 00 Four character 2-byte address field; hexidecimal address 0x0000, where the data which follows is to be loaded. o 28 5F 24 5F 22 12 22 6A 00 04 24 29 00 08 23 7C Sixteen character pairs representing the actual binary data. o 2A The checksum. The second and third S1 records each contain 0x13 (19) character pairs and are ended with checksums of 13 and 52, respectively. The fourth S1 record contains 07 character pairs and has a checksum of 92. The S5 record is comprised as follows: o S5 S-record type S5, indicating it is a count record indicating the number of S1 records o 03 Hexadecimal 03 (decimal 3), indicating that three character pairs follow. o 00 04 Hexadecimal 0004 (decimal 4), indicating that there are four data records previous to this record. o F8 The checksum. The S9 record is comprised as follows: o S9 S-record type S9, indicating it is a termination record. o 03 Hexadecimal 03 (decimal 3), indicating that three character pairs follow. o 00 00 The address field, hexadecimal 0 (decimal 0) indicating the starting execution address. o FC The checksum. NOTES
o There isn't any evidence that Motorola ever made use of the header information within the data field of the S0 record, as described above. This may have been used by some third party vendors. o The Unix manual page on S-records is the only place that a 78-byte limit on total record length or 64-byte limit on data length is documented. These values shouldn't be trusted for the general case. o The count field can have values in the range of 0x3 (2 bytes of address + 1 byte checksum = 3, a not very useful record) to 0xff; this is the count of remaining character pairs, including checksum. o If you write code to convert S-Records, you should always assume that a record can be as long as 514 (decimal) characters in length (255 * 2 = 510, plus 4 characters for the type and count fields), plus any terminating character(s). That is, in establishing an input buffer in C, you would declare it to be an array of 515 chars, thus leaving room for the terminating null character. SEE ALSO
gpsd(8), gps(1), libgps(3), libgpsd(3), gpsfake(1). gpsprof(1). AUTHOR
From an anonymous web page, itself claiming to have been derived from an old Unix manual page. Now maintained by the GPSD project, which added endianness clarifications. The GPSD Project 15 Jul 2005 SREC(5)
All times are GMT -4. The time now is 07:40 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy