Visit The New, Modern Unix Linux Community

Fixed width file issue and resolutions

Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Answers to Frequently Asked Questions Fixed width file issue and resolutions
# 1  
Fixed width file issue and resolutions

I have a fixed width file with no separators , but the file has JUNK characters in it and i know how to remove it. but in few cases these junk characters having created a mess by splitting single row into 2 or 3 rows.
I need to put them back to a single line like rest of the rows. This fixed width file as rows which is of length 1-484.

The only way to identify that the row is spilt into 2 or 3 rows is by its length only and that too is not consistent.

Example :
Row expected :
0401000000030020170628000000710000366400201706280002750035*T000100N▒101892928550383900000009201 00000000000008402017062800000020  0000006435000000000000000840A1E098D09D9279BE4000561510A00003220000000000000000000000FF                            05A17DD6318008000048000        83337602E0F8C80002420300    00000000000000                                                   0000002278BF8AC19546B1EE0012            A0000000042203                  A0000000042203                  R 000000199551

Row with the issue : CASE1 :
T000100N▒101909856755446700000018201 00000000000008402017062800000000  00000067540000000000000008402ED730917E9D1DC040000B0810A04003240000000000000000000000FF                            083426C5739000000001000        83337602E0F8C800021E0300    00000000000000                                                   00000022F85D9594E4FC84E2031A            A0000000041010                  A0000000041010                  R 000000199442

Row with the issue : CASE2 :

0000034655000000000000000840A6258473ADBF88E24000740110A04001220000000000000000000000FF                            06E09E43739000000008000        88131669E0F8C800021E0300    00000000000000                                                   00000022B050CC3E5A75C2E90012            A0000000041010                  A0000000041010                  R 000002274242

Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 07-19-2017 at 04:21 AM.. Reason: Added CODE tags.
# 2  
Welcome to the forum.

Why don't you post the method / script / code that you use to remove the junk characters? It might be better to avoid the errors in the first place than to run a second code to deal with them.
# 3  
Welcome santoshkm,

I have a few to questions pose in response first:-
  • What have you tried so far?
  • What output/errors do you get?
  • Can you show us the data piped through od -x so we can see the character codes? Having the original data on the next line would be useful too.
  • What OS and version are you using?
  • What are your preferred tools? (C, shell, perl, awk, etc.)
  • What logical process have you considered? (to help steer us to follow what you are trying to achieve)
Most importantly, What have you tried so far?

There are probably many ways to achieve most tasks, so giving us an idea of your style and thoughts will help us guide you to an answer most suitable to you so you can adjust it to suit your needs in future.

We're all here to learn and getting the relevant information will help us all.

Thanks, in advance,

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #259
Difficulty: Easy
Steve Jobs was born in San Francisco, California, and put up for adoption.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk issue splitting a fixed-width file containing line feed in data

Hi Forum. I have the following script that splits a large fixed-width file into smaller multiple fixed-width files based on input segment type. The main command in the script is: awk -v search_col_pos=$search_col_pos -v search_str_len=$search_str_len -v segment_type="$segment_type"... (8 Replies)
Discussion started by: pchang
8 Replies

2. Shell Programming and Scripting

Replace using awk on fixed width file.

All, I used to use following command to replace specific location in a fixed width file. Recently looks like my command stopped working as intended. We are on AIX unix. awk 'function repl(s,f,t,v) { return substr(s,1,f-1) sprintf("%-*s", t-f+1, v) substr(s,t+1) } NR<=10 {... (3 Replies)
Discussion started by: pinnacle
3 Replies

3. Shell Programming and Scripting

Alter Fixed Width File

Thank u so much .Its working fine as expected. ---------- Post updated at 03:41 PM ---------- Previous update was at 01:46 PM ---------- I need one more help. I have another file(fixed length) that will get negative value (ex:-00000000003000) in postion (98 - 112) then i have to... (6 Replies)
Discussion started by: vinus
6 Replies

4. UNIX for Dummies Questions & Answers

Length of a fixed width file

I have a fixed width file of length 53. when is try to get the lengh of the record of that file i get 2 different answers. awk '{print length;exit}' <File_name> The above code gives me length 50. wc -L <File_name> The above code gives me length 53. Please clarify on... (2 Replies)
Discussion started by: Amrutha24
2 Replies

5. Shell Programming and Scripting

Fixed width file comparision not working

when i used diff/cmp/cat -v commands i am getting the difference cmp command cmp -l file1 file2 |head -1 1300 15 10 Manually checked records. record length and data matched. diff file1 file2 3C3 <record information Manually checked records. record length and data matched.... (4 Replies)
Discussion started by: onesuri
4 Replies

6. Shell Programming and Scripting

Comparing two fixed width file

Hi Guys I am checking the treads to get the answer but i am not able to get the answer for my question. I have two files. First file is a pattern file and the second file is the file i want to search in it. Output will be the lines from file2. File1: P2797f12af 44751228... (10 Replies)
Discussion started by: anshul_er
10 Replies

7. Shell Programming and Scripting

Fixed-Width file from Oracle

Hi All, I have created a script which generates FIXED-WIDTH file by executing Oracle query. SELECT RPAD(NVL(col1,CHR(9)),20)||NVL(col2,CHR(9))||NVL(col3,CHR(9) FROM XYZ It generates the data file with proper alignment. But if same file i transfer to windows server or Mainframe... (5 Replies)
Discussion started by: Amit.Sagpariya
5 Replies

8. Shell Programming and Scripting

summing up the fields in fixed width file

Hi, I have a fixed width file with some records as given below: " 1000Nalsdjflj243324jljlj" "-0300Njfowjljl309933fsf" " 0010Njsfsjklj342344fsl" I want to sum-up first field values(i.e from 2nd character to 6th character)of each record. so for the above file i want to add (1000 - 300+... (2 Replies)
Discussion started by: srilaxmi
2 Replies

9. UNIX Desktop Questions & Answers

Help with Fixed width File Parsing

I am trying to parse a Fixed width file with data as below. I am trying to assign column values from each record to variables. When I parse the data, the spaces in all coumns are dropped. I would like to retain the spaces as part of the dat stored in the variables. Any help is appreciated. I... (4 Replies)
Discussion started by: sate911
4 Replies

10. UNIX for Dummies Questions & Answers

Fixed Width file using AWK

I am using the following command at the Unix prompt to make my 'infile' into a fixed width file of 100 characters. awk '{printf "%-100s\n",$0}' infile > outfile However, there are some records with a special character "" These records are using 3 characters in place of one and my record... (2 Replies)
Discussion started by: alok.benjwal
2 Replies

Featured Tech Videos