Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Sorting file with CRLF within field, RS=$

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Sorting file with CRLF within field, RS=$
# 1  
Old 01-29-2016
Sorting file with CRLF within field, RS=$
OK below is what my sample file looks like. I need to sort by the Primary Key ie: {1:F01SAESVAV0AXXX0466020126} in the first record. Record seperator is $.

I tried sort, but it completely messes it up. I am thinking I will need to use something like awk which understands the record seperator and allows me to substring or something to sort by PK. Any assistance is greatly appreciated.

Source File
Code:
{1:F01SAESVAV0AXXX0466020126}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$
{1:F01SAESVAV0AXXX0466020123}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$
{1:F01SAESVAV0AXXX0466020121}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$

Output I want:
Code:
{1:F01SAESVAV0AXXX0466020121}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$
{1:F01SAESVAV0AXXX0466020123}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$
{1:F01SAESVAV0AXXX0466020126}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$

# 2  
Old 01-29-2016
Are these DOS files? UNIX and Linux system utilities generally expect <newline> (not CR/LF) line terminators?

What operating system, shell, and version of awk are you using?
# 3  
Old 01-29-2016
Operating System: GNU/Linux
AWK Version: GNU Awk 3.1.7

I am not sure yet as to the source system whether it is Unix or DOS but I know that End of Message is denoted by: $. File will be processed on linux.
# 4  
Old 01-29-2016
Got Perl?

Code:
perl -ne 'BEGIN{$/="\$\n"}; ($i) = /^({[^}]*})/ and $r{$i} = $_; END{for(sort keys %r){print $r{$_}}}' alfredo123.file

Code:
{1:F01SAESVAV0AXXX0466020121}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$
{1:F01SAESVAV0AXXX0466020123}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$
{1:F01SAESVAV0AXXX0466020126}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$

Last edited by Aia; 01-29-2016 at 07:14 PM.. Reason: Minor misspell
This User Gave Thanks to Aia For This Post:
alfredo123
# 5  
Old 01-30-2016
Would this do?
Code:
awk '/^{1:/ {LAST=$1} {print LAST, NR, $0}' FS="}" OFS="}" file | sort -t"}" -k1,1 -k2,2n | cut -d"}" -f3-

This User Gave Thanks to RudiC For This Post:
alfredo123
# 6  
Old 02-01-2016
awk command worked.
Sincere appreciation for the response provided. Unfortunately I do not have Perl so the perl option would not work for me. The awk command provided worked but I would certainly like to understand what is going on here:

Code:
awk '/^{1:/ {LAST=$1} {print LAST, NR, $0}' FS="}" OFS="}" file | sort -t"}" -k1,1 -k2,2n | cut -d"}" -f3-

Also, it seems like the Unique key for sort would be the first 2 tags instead of just the first one and the $ would be at the start of every new row instead of at the end of each row. If I get an understanding of what is going on here, maybe I can implement the sort based on 2 tags and identify new record based on dollar.

Unique Key:
Code:
{1:F01SAESVAV0AXXX0466020121}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}

So my source file is now going to look like this:
Code:
{1:F01SAESVAV0AXXX0466020126}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}
${1:F01SAESVAV0AXXX0466020123}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}
${1:F01SAESVAV0AXXX0466020121}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$

# 7  
Old 02-01-2016
Sorry, I don't understand what you are saying. When applying my proposal to your sample input, I get
Code:
{1:F01SAESVAV0AXXX0466020121}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$
{1:F01SAESVAV0AXXX0466020123}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$
{1:F01SAESVAV0AXXX0466020126}{2:O1011538070522LRLRXXXX4A0700005910650705221739N}{3:{108:MT101 001 OF 019}}{4:
:20:00028
:28D:1/1
:50H:/VTB.2003.02
19Apr2002
:30:020419
:21:x
:32B:USD1,
:50L:x
:59:/x
x
:71A:OUR
-}{5:{MAC:00000000}{CHK:24857F4599E7}{TNG:}}{S:{SAC:}{COP:P}}$

which is in the correct sort order, and the $- signs are in the correct spot.
This User Gave Thanks to RudiC For This Post:
alfredo123
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sorting based on the second field

Oracle Enterprise Linux 6 This is my file. Two fields separated by space $ cat testfile.txt MARCH9 MARCH4 MARCH1 MARCH5 MARCH2 MARCH326 MARCH821 MARCH7 MARCH6 MARCH2 $ $ The following numeric sort, based on the first field's 6th character works as expected. $ $ sort -n -k 1.6... (7 Replies)
Discussion started by: John K
7 Replies

2. Shell Programming and Scripting

sorting based on a field

the below is sorted as it is. the fields that i'm interested in are the 4th and 5th field. i want to sort the based on the 4th field. my past attempt to do this was to do something like this: awk '{print $4}'| awk '{print $1":"$2}' datafile | sort | uniq however, if i do that, i lose... (2 Replies)
Discussion started by: SkySmart
2 Replies

3. Shell Programming and Scripting

Sorting Date Field with Sort -k :/

SOLVED : (17 Replies)
Discussion started by: Glitch100
17 Replies

4. UNIX for Dummies Questions & Answers

Sorting data in file based on field in another file

Hi, I have two files, one of which I would like to sort based on the order of the data in the second. I would like to do this using a simple unix statement. My two files as follows: File 1: 12345 1 2 2 2 0 0 12349 0 0 2 2 1 2 12350 1 2 1 2 2 2 . . . File2: 12350... (3 Replies)
Discussion started by: kasan0
3 Replies

5. Programming

Sorting a multidimensional vector by a specific field.

In some cases I would like to sort by index, in some cases by color and in some cases by Callsign. Can this be done? :D vector< vector<string> > table; vector<string> row; row.push_back("1");row.push_back("green");row.push_back("alpha"); table.push_back(row);... (0 Replies)
Discussion started by: sepoto
0 Replies

6. Shell Programming and Scripting

Sorting on two fields time field and number field

Hi, I have a file that has data in it that says 00:01:48.233 1212 00:01:56.233 345 00:09:01.221 5678 00:12:23.321 93444 The file has more line than this but i just wanted to put in a snippet to ask how I would get the highest number with time stamp into another file. So from the above... (2 Replies)
Discussion started by: pat4519
2 Replies

7. Shell Programming and Scripting

Sorting file by a field, and then by another field.

Hi all, Sorry the title is a mess, but did not find a better description at the time. So here is my problem: I have an input file: 8:Mass40s -- 0 48:Mass40s -- 0 67:Mass40s -- 0 86:Mass40s -- 0 105:Mass40s -- 0 9:Mass -- 1 49:Mass -- 86... (5 Replies)
Discussion started by: Alexis Duarte
5 Replies

8. Shell Programming and Scripting

Issue with a file that contains CRLF

I have a nawk that reads in a log file and outputs a file that matches my search. IFS=" " while read record do `echo $record | nawk 'BEGIN { FS=" " } { type_record=substr($0, 1, 1); if (... (14 Replies)
Discussion started by: Pablo_beezo
14 Replies

9. Shell Programming and Scripting

Sorting on last field

Hello I am trying to sort on the last field and it seems to have lost ideas on what to do. the file am sorting looks like this Tan\da\1223 hey\1234 two\three\think\4579 i want to sort on the last fields (1223, 1234 and 4579). thank you (2 Replies)
Discussion started by: ganiel24
2 Replies

10. Programming

how to add CRLF support for CSV file generated in unix

Helo, my server side system is running on a redhat linux o.s. I have c program on the server which export list file into CSV format. now, I want that my program on server side is able to add support for CRLF(carriage return Line feed)into csv file format. so how do i write C program whcih... (3 Replies)
Discussion started by: amitpansuria
3 Replies
Login or Register to Ask a Question