Alternative to sort -ur +1 required - Page 2 | Unix Linux Forums | UNIX for Advanced & Expert Users

  Go Back    


UNIX for Advanced & Expert Users Expert-to-Expert. Learn advanced UNIX, UNIX commands, Linux, Operating Systems, System Administration, Programming, Shell, Shell Scripts, Solaris, Linux, HP-UX, AIX, OS X, BSD.

Alternative to sort -ur +1 required

UNIX for Advanced & Expert Users


Tags
sort

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #8  
Old 10-08-2012
Don Cragun's Avatar
Don Cragun Don Cragun is online now Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 2 September 2014, 9:55 PM EDT
Location: San Jose, CA, USA
Posts: 4,506
Thanks: 177
Thanked 1,512 Times in 1,284 Posts
A straight translation of
Code:
sort -ur +1

using the +w and -x options to specify sort keys to using -k y,z options to specify sort keys is
Code:
sort -ur -k2

When your input lines are a numeric string representing a timestamp followed by a combination of one or more space and tab characters followed by "other text" ending with the line's terminating newline, this command will sort "other text" in reverse order and discard all but one line with identical values of "other text". If two or more lines have the same "other text" but different timestamps, which timestamp will be kept is unspecified.
Sponsored Links
    #9  
Old 10-09-2012
Mike Smith Mike Smith is offline
Registered User
 
Join Date: Oct 2012
Last Activity: 30 January 2013, 6:24 AM EST
Posts: 6
Thanks: 0
Thanked 0 Times in 0 Posts
Tried -k2,2 and it binned loads of old entries which I wanted!

Say I have this data, the highest number (newest) one will always be at the top and I only want to keep the top one as everything right of, and including, column 2 is the same (nothing has changed since last sweep).
1349455502 ygtr-1b:3/1/19 10/100/Gig Ethernet SFP
1349246545 ygtr-1b:3/1/19 10/100/Gig Ethernet SFP

However if we get the following then I want both entries retaining
1349455502 ygtr-1b:3/1/19 10/100/Gig Ethernet SFP
1349246545 ygtr-1b:3/1/19 Old customer name typed in here

-k2,2 seems to ONLY consider column 2 which is no good based on my first post's criteria.

This is tricky!
Sponsored Links
    #10  
Old 10-09-2012
Don Cragun's Avatar
Don Cragun Don Cragun is online now Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 2 September 2014, 9:55 PM EDT
Location: San Jose, CA, USA
Posts: 4,506
Thanks: 177
Thanked 1,512 Times in 1,284 Posts
Quote:
Originally Posted by Mike Smith View Post
Tried -k2,2 and it binned loads of old entries which I wanted!

Say I have this data, the highest number (newest) one will always be at the top and I only want to keep the top one as everything right of, and including, column 2 is the same (nothing has changed since last sweep).
1349455502 ygtr-1b:3/1/19 10/100/Gig Ethernet SFP
1349246545 ygtr-1b:3/1/19 10/100/Gig Ethernet SFP

However if we get the following then I want both entries retaining
1349455502 ygtr-1b:3/1/19 10/100/Gig Ethernet SFP
1349246545 ygtr-1b:3/1/19 Old customer name typed in here

-k2,2 seems to ONLY consider column 2 which is no good based on my first post's criteria.

This is tricky!
It isn't tricky at all. You can't say that the first field doesn't matter when throwing away duplicates and at the same time say that you want the highest value for the first field to be the one that is kept when throwing away duplicates. If you care which duplicate is kept, you don't think they are duplicates. If you want both, you have to do it in two separate steps:
  1. sort in reverse order with the data from the start of field 2 to the end of the line as your primary sort key and the numeric value in field 1 as your secondary sort key, and then
  2. on a second pass, discard or ignore the second and subsequent lines that match (including field separators other than the separator between the 1st and 2nd fields) from the start of the 2nd field to the end of the line.

So why does it now matter which timestamp is kept when throwing away duplicates if it didn't matter when you were running this application on Solaris 8?
    #11  
Old 10-09-2012
DGPickett DGPickett is offline Forum Advisor  
Registered User
 
Join Date: Oct 2010
Last Activity: 29 August 2014, 5:00 PM EDT
Location: Southern NJ, USA (Nord)
Posts: 4,409
Thanks: 8
Thanked 539 Times in 517 Posts
To do it in one pass, sort by user and then by time, and only print the first every time the user changes. To do it in one pass without sorting, store the time using a string addressable vector keyed to the user and overwrite it for later times.
Sponsored Links
    #12  
Old 10-22-2012
Mike Smith Mike Smith is offline
Registered User
 
Join Date: Oct 2012
Last Activity: 30 January 2013, 6:24 AM EST
Posts: 6
Thanks: 0
Thanked 0 Times in 0 Posts
@Don

Since this works perfectly on a different box and occassionally on this one I was hoping that I'd not have to reinvent this thing to fix my problem.

However I think what you're describing is basic enough for me to manage.

sort -k2 -k1,1 newest-data one-month-data archive-data | sort -ur > output

Sound about right?

And yes if there are two identical lines with just the timestamp being different then I'd like to keep the biggest number (newest)
Sponsored Links
    #13  
Old 10-22-2012
Don Cragun's Avatar
Don Cragun Don Cragun is online now Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 2 September 2014, 9:55 PM EDT
Location: San Jose, CA, USA
Posts: 4,506
Thanks: 177
Thanked 1,512 Times in 1,284 Posts
Quote:
Originally Posted by Mike Smith View Post
@Don

Since this works perfectly on a different box and occassionally on this one I was hoping that I'd not have to reinvent this thing to fix my problem.

However I think what you're describing is basic enough for me to manage.

sort -k2 -k1,1 newest-data one-month-data archive-data | sort -ur > output

Sound about right?

And yes if there are two identical lines with just the timestamp being different then I'd like to keep the biggest number (newest)
No! Absolutely not! Never! If you are feeding the data through sort twice, the first sort has absolutely no effect (unless you use a -u option in the 1st sort to discard some data and you have already learned that you can't use -u in the 1st sort). The command line you're suggesting:

Code:
sort -k2 -k1,1 newest-data one-month-data archive-data | sort -ur > output

is functionally equivalent to:

Code:
sort -ur newest-data one-month-data archive-data > output

Both sort commands sort the entire set of input lines according to the sort key specified by that sort command.

You still haven't explained why it matters what the timestamp is on lines that are otherwise identical. The commands that you had on Solaris 8 randomly kept one of the lines that matched from the start of field 2 to the end of the line. As stated before the command:

Code:
sort -ur -k2  newest-data one-month-data archive-data

will do what would have happened on Solaris 8 with your current data. If that isn't sufficient, the first step I stated for you in message #10 in this thread can be implemented using:

Code:
sort -k2r -k1nr,1 newest-data one-month-data archive-data

but you will need to write another program that reads the data written by the above sort and throws away all but the 1st line of each set of lines that are identical from the start of field 2 to the end of the line. The program that will do this is NOT sort. It is probably an awk script that compares the substring starting at the first character of column 2 and continuing to the end of the line for adjacent lines and prints $0 for the 1st line in each matching set. (Note that this is not the same as comparing fields $2 to $NF because differences in field separators matter in the first case, but are ignored in the second case.)
Sponsored Links
    #14  
Old 10-23-2012
DGPickett DGPickett is offline Forum Advisor  
Registered User
 
Join Date: Oct 2010
Last Activity: 29 August 2014, 5:00 PM EDT
Location: Southern NJ, USA (Nord)
Posts: 4,409
Thanks: 8
Thanked 539 Times in 517 Posts
I believe that 'sort -u' saves just the first occurrance of the unique key, so you sort first non-unique to get the right first record saved.

However, I agree it is a bit of a shame to use so much storage and processing when you could just tuck them in an associative array and overwrite any old values, especially in cases where the data starts out sorted in some relevant way. The only drawback is that the speed of shell operations might be a drag on big volume. You can scale up sort in parallel using pipes and sort -m, but for the unsorted lookup solution at machine speed, C++ or at least JAVA can work a hash table faster, and you can pre-size the hash table big enough to get good use of RAM and VM in even a 32 bit app. I like big powers of 2, since a modulus of the hash becomes a lower bit mask. Empty hash table entries are just pointers in an array, 4 or 8 bytes cost each, which is pretty cheap, and does no harm for smaller data sets! Hash beats tree for query and churn speed, but tree does provide sorted output and scales more automatically. Linear hash (tables in power of two sizes that can double the hash table size for congested buckets) has a better dynamic scaling, but slower query and churn than straight hash. I have not found a lot of hash implementations that reveal they are linear.
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Script to sort the files and append the extension .sort to the sorted version of the file pankaj80 UNIX for Advanced & Expert Users 3 06-07-2011 09:28 AM
How to insert alternative columns and sort text from first column to second? Unilearn UNIX for Dummies Questions & Answers 7 07-10-2010 06:22 AM
Getting required fields from a test file in required fromat in unix rdhanek Shell Programming and Scripting 7 07-22-2009 11:35 AM
sort out the required data imas UNIX for Advanced & Expert Users 6 04-24-2009 04:46 AM
Script required to get a required info from file. Pls. help me. ntgobinath Shell Programming and Scripting 2 05-31-2008 08:34 AM



All times are GMT -4. The time now is 10:08 PM.