Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: awk to extract incorrect fixed length records
Top Forums Shell Programming and Scripting awk to extract incorrect fixed length records Post 302345778 by methyl on Thursday 20th of August 2009 08:02:38 AM
Old 08-20-2009
awk to extract incorrect fixed length records
I have a number of unix text files containing fixed-length records (normal unix linefeed terminator) where I need to find odd records which are an incorrect length.
The data is not validated and records can contain odd backslash characters and control characters which makes them awkward to process in shell. Files are unlikely to exceed 1000 records, so efficiency is not important. Each file has a different fixed length record size.

I am looking for an awk program which can read the length of the first record of a file, then output only those records which are of a different length to the first record. The program does not need to cater for the first record being wrong.

This post works beautifully, but you need to know the length of the record in advance.

https://www.unix.com/shell-programmin...th-record.html
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

creating a fixed length output from a variable length input

Is there a command that sets a variable length? I have a input of a variable length field but my output for that field needs to be set to 32 char. Is there such a command? I am on a sun box running ksh Thanks (2 Replies)
Discussion started by: r1500
2 Replies

2. Shell Programming and Scripting

fixed length fields in awk

I am trying to display df -h command out in proper format, how can I display each field of each record in a fixed length. (2 Replies)
Discussion started by: roopla
2 Replies

3. Shell Programming and Scripting

Awk with fixed length files

Hi Unix Champs, I want to awk on a fixed length file. Instead if the file was a delimited file, then I could have used -F and then could have easily done manipulation on the fields. How do i do the same in case of fixed length file? Thanks in Advance. Regards. (7 Replies)
Discussion started by: c2b2
7 Replies

4. Shell Programming and Scripting

Fixed Length records- Korne Shell Program.

Hi, I need some help regarding in writing a Korne shell script, in determining the fixed length records in a data file. We have already utility in place, which does this work. The Code for this is as below. In the below $1 is the parameter passed to the script, which is the data file name. ... (4 Replies)
Discussion started by: nrajesh_2009
4 Replies

5. Shell Programming and Scripting

Awk - Working with fixed length files

OK I am somewhat new to UNIX programming please see what you can do to help. I have a flat file that is a fixed length file containing different records based on the 1st character of each line. The 1st number at the beginning of the line is the record number, in this case it's record #1. I... (3 Replies)
Discussion started by: ambroze
3 Replies

6. Shell Programming and Scripting

Using awk to search variable length records

New to awk and need some help. I have a script that I would like to make more compact. I want to read a file and grab every field, from every record, except the last field. The records are variable length and have varying number of fields. A record will have at least two fields, but can have... (9 Replies)
Discussion started by: synergy_texas
9 Replies

7. Shell Programming and Scripting

Need awk script to compare 2 fields in fixed length file.

Need a script that manipulates a fixed length file that will compare 2 fields in that file and if they are equal write that line to a new file. i.e. If fields 87-93 = fields 119-125, then write the entire line to a new file. Do this for every line in the file. After we get only the fields... (1 Reply)
Discussion started by: Muga801
1 Replies

8. Shell Programming and Scripting

Unix sort for fixed length columns and records

I was trying to use the AIX 6.1 sort command to sort fixed-length data records, sorting by specific columns only. It took some time to figure out how to get it to work, so I wanted to share the solution. The sort man page wasn't much help, because it talks about field delimeters (default space... (1 Reply)
Discussion started by: CheeseHead1
1 Replies

9. Shell Programming and Scripting

awk to print fixed length columns to right side

Hi, I am in a situation to print the message on a column, where the each line starting position should be same. For example code: HOSTNAME1="1.2.3.4.5.6.7" TARGET_DIR="/tmp" echo "HOSTNAME1:" "$HOSTNAME1" | awk -v var="Everyone" '{len=55-length;printf("%s%*s\n",$0,len,var)}' echo... (4 Replies)
Discussion started by: tprabhaker
4 Replies

10. Shell Programming and Scripting

Splitting fixed length file using awk

Hi, I need to split a fixed length file of 160 characters based on value of a column. Example: ABC 456780001 DGDG SDFSF BCD 444440002 SSSS TTTTT ABC 777750003 HHHH UUUUU THH 888880001 FFFF LLLLLL HHH 999990002 GGGG OOOOO I need to split this file on basis of column from... (7 Replies)
Discussion started by: Neelkanth
7 Replies

LEARN ABOUT DEBIAN

srec_brecord

srec_brecord(5) 						File Formats Manual						   srec_brecord(5)

NAME

       srec_brecord - Freescale MC68EZ328 Dragonball bootstrap record format

DESCRIPTION

       This data format is understood by Freescale MC68EZ328 Dragonball series processors on their internal UART.

   Lines
       Each  line  contains  hexadecimal  data,  each  byte represented by two hexadecimal nybbles in upper case.  Characters not in this set, but
       larger than 0x30 (e.g. lower case) will be ignored, less than 0x30 (e.g. CR or LF) are considered record terminators.  Comments	are  prob-
       lematic; don't try this at home.

   Fields
       Each  line  contains  a	4-byte address (big endian), a 1-byte length-and-mode, and then data bytes as dictated by the length.  There is no
       checksum.  A zero length record is an execution start address record, non-zero length records are data.

						   +--+---+---+---+---+---+---+---+---+----+-----+---+
						   |1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | ... | n |
						   +--+---+---+---+---+---+---+---+---+----+-----+---+
						   |	       Address		  | Length |  Data   |
						   +------------------------------+--------+---------+
       The length-and-mode byte is formatted as follows:

							     +--+---+---+---+---+---+---+---+
							     |7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
							     +--+---+---+---+---+---+---+---+
							     |Mode  | R |      Length	    |
							     +------+---+-------------------+
       Mode    These bits are ignored by SRecord in input (00 = bytes, 01 = half words, 10 is reserved, 11 = long words).  These bits  are  always
	       zero on output by SRecord.

       R       This bit indicates a data read rather than a data write; SRecord does not accept input files with this bit set, and will not set it
	       on output.

       Length  The length of the records data bytes.  It does not include the address or length bytes.	The maximum payload  of  a  record  is	31
	       bytes of data.

   Size Multiplier
       In general, binary data will expand in sized by at least 2.35 times when represented with this format.

EXAMPLE

       Here is an example b-record format file.  It contains the data "Hello, World" to be loaded at address 0.
	      000000000D48656C6C6F2C20576F726C640A

SEE ALSO

       http://www.freescale.com/files/32bit/doc/ref_manual/MC68VZ328UM.pdf

COPYRIGHT

       srec_cat version 1.58
       Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Peter Miller

       The srec_cat program comes with ABSOLUTELY NO WARRANTY; for details use the 'srec_cat -VERSion License' command.  This is free software and
       you are welcome to redistribute it under certain conditions; for details use the 'srec_cat -VERSion License' command.

AUTHOR

       Peter Miller   E-Mail:	pmiller@opensource.org.au
       //*		 WWW:	http://miller.emu.id.au/pmiller/

Reference Manual						      SRecord							   srec_brecord(5)
All times are GMT -4. The time now is 01:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy