Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Sorting data - awk or sort or both

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Sorting data - awk or sort or both
# 1  
Old 08-18-2014
Sorting data - awk or sort or both
data:

Code:
C812F5C9B   0818053014 P S SYSPROC        SOFTWARE PROGRAM ABNORMALLY TERMINATED
C812F5C9B   0818054514 P S SYSPROC        SOFTWARE PROGRAM ABNORMALLY TERMINATED
C812F5C9B   0818060014 P S SYSPROC        SOFTWARE PROGRAM ABNORMALLY TERMINATED
C812F5C9B   0818061514 P S SYSPROC        SOFTWARE PROGRAM ABNORMALLY TERMINATED
C812F5C9B   0818063014 P S hdisk45             SOFTWARE PROGRAM ABNORMALLY TERMINATED

i have a huge data that has an output similar to the above.

i'm using the following command to try to get rid of any duplicates:

Code:
sort -k 1,1 -k 2,2 -k3,3 -u

not enough dups are being eliminated. so i was wondering if you guys have a better approach to this.

i would like to sort and unique by the first 6 characters of the 2nd field.

the numbers in the second field mean:

Code:
let's use 0818063014 as an example:

08 = month
18 = day
06 = hour
30 = minute
12 = year

is what i'm trying to do possible?
# 2  
Old 08-18-2014
Try this (untested):

Code:
awk '!dupe[substr($2,1,6)]++' file

Sorry missed the obvious bit about sorting! This should remove duplicates only, it can then be piped to sort.
This User Gave Thanks to pilnet101 For This Post:
SkySmart
# 3  
Old 08-18-2014
Quote:
Originally Posted by pilnet101
Try this (untested):

Code:
awk '!dupe[substr($2,1,6)]++' file

Sorry missed the obvious bit about sorting! This should remove duplicates only, it can then be piped to sort.
this actually may work.

i ran it, and it appears to be getting rid of more than i want. but i feel with a little more massaging, i can get it to do what i want.

here's what im running:

Code:
sort -k 1,1 -k 2,2 -k 3,3 -u file | awk '!dupe[substr($2,1,6)]++'

# 4  
Old 08-19-2014
Try
Code:
sort -u -k2.1,2.9 file
C812F5C9B   0818053014 P S SYSPROC        SOFTWARE PROGRAM ABNORMALLY TERMINATED
C812F5C9B   0818060014 P S SYSPROC        SOFTWARE PROGRAM ABNORMALLY TERMINATED

2.9 is the key's stop position, as
Quote:
If neither -t nor -b is in effect, characters in a field are
counted from the beginning of the preceding whitespace.
(man sort)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Is this sort working as it is supposed to be? A bit confused whether it is sorting correctly or not

Hi, Below is the sample file: $ cat x.txt MDSYS|OGIS_GEOMETRY_COLUMNS|TABLE MDSYS|OGIS_SPATIAL_REFERENCE_SYSTEMS|TABLE MDSYS|SDO_IDX_TAB_SEQUENCE|SEQUENCE MDSYS|SDO_PREFERRED_OPS_USER|TABLE MDSYS|SDO_ST_TOLERANCE|TABLE MDSYS|SDO_TOPO_DATA$|TABLE MDSYS|SDO_TOPO_RELATION_DATA|TABLE... (4 Replies)
Discussion started by: newbie_01
4 Replies

2. Shell Programming and Scripting

Sorting a html file with an external sort order

I am working on a web-concordance of Old Avestan and my concordance has produced a HTML file The sort deployed by the HTML file is not something which we normally use. I have tried my best to force a sort within the concordance itself, but the sort order does not work. I am giving below the sort... (6 Replies)
Discussion started by: gimley
6 Replies

3. OS X (Apple)

Sorting scientific numbers with sort

Hey everybody, I'm trying to sort scientific numbers in a descending order using the command sort -gr <file>. It works fine on a Linux-Server, but doesn't on my macbook pro with OS X 10.10.3 (Yosemite). I tried to sort the following: 6.38e-10 6.38e-10 1.80e-11 1.00e-10 1.48e-12 And... (9 Replies)
Discussion started by: plebs
9 Replies

4. Shell Programming and Scripting

Help: Sorting the numbers without using sort command

Hi, I've got two arrays 1 3 5 7 2 4 6 8 and i need to write a shell script to get the output 1 2 3 4 5 6 7 8 without using sort or bubble sort. (1 Reply)
Discussion started by: web2moha
1 Replies

5. UNIX for Dummies Questions & Answers

Sorting binary files using UNIX sort

Hi, Can i sort binary files using unix sort ? (4 Replies)
Discussion started by: AmbikaValagonda
4 Replies

6. Shell Programming and Scripting

Sorting Date Field with Sort -k :/

SOLVED : (17 Replies)
Discussion started by: Glitch100
17 Replies

7. UNIX for Dummies Questions & Answers

sorting numbers with sort -n

Looking for help for sort, I learned that for sorting numbers I use: sort -n but it seems that that is not enough when you have numbers like 0.2000E+7 for example, sort -n will not worry about the E+7 part, and will just sort the numbers like 0.2000. Exapmle: cat example.txt .91000E+07... (9 Replies)
Discussion started by: cosmologist
9 Replies

8. Shell Programming and Scripting

Sorting the data right justified with through awk.

Dear all, I have a inputfile like some what like this - 1000.98651 1000.96696 999.98904 991.66864 986.51829 986.49467 17.44122 16.74039 16.74021 10.92725 desired output 1000.98651 1000.96696 0999.98904 (4 Replies)
Discussion started by: admax
4 Replies

9. Shell Programming and Scripting

sort: 0653-657 A write error occurred while sorting.

Hi I am trying to sort a file of 88075743B size. I am doing some processing on the file and after the processing is done; I get 2 files temp1 and temp2. I need to combine both these files as one and this final file should be sorted on fields 1 and 2. Space is the delimiter between fields. Record... (2 Replies)
Discussion started by: diksha2207
2 Replies

10. Shell Programming and Scripting

Newbie Awk data sorting

Hi guys, just started using awk here. I've got a file called a.txt which contains a load of numbers on each line e.g. 35 232 654 1 9 4 I want to learn how to do three things: 1. Find the minimum 2. Find the maximum 3. Find the average I want to learn how to do this using awk.... (5 Replies)
Discussion started by: i_am_a_robot
5 Replies
Login or Register to Ask a Question