Sponsored Content
Top Forums UNIX for Advanced & Expert Users Alternative to sort -ur +1 required Post 302719867 by DGPickett on Tuesday 23rd of October 2012 09:54:17 AM
Old 10-23-2012
I believe that 'sort -u' saves just the first occurrance of the unique key, so you sort first non-unique to get the right first record saved.

However, I agree it is a bit of a shame to use so much storage and processing when you could just tuck them in an associative array and overwrite any old values, especially in cases where the data starts out sorted in some relevant way. The only drawback is that the speed of shell operations might be a drag on big volume. You can scale up sort in parallel using pipes and sort -m, but for the unsorted lookup solution at machine speed, C++ or at least JAVA can work a hash table faster, and you can pre-size the hash table big enough to get good use of RAM and VM in even a 32 bit app. I like big powers of 2, since a modulus of the hash becomes a lower bit mask. Empty hash table entries are just pointers in an array, 4 or 8 bytes cost each, which is pretty cheap, and does no harm for smaller data sets! Hash beats tree for query and churn speed, but tree does provide sorted output and scales more automatically. Linear hash (tables in power of two sizes that can double the hash table size for congested buckets) has a better dynamic scaling, but slower query and churn than straight hash. I have not found a lot of hash implementations that reveal they are linear.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

sort out the required data

Hi All, I have a file 1.txt which has the duplicate dns entries as shown: Name: 000f9fbc6738.net.in|Addresses: 10.241.66.169, 10.84.2.222,212.241.66.170 Name: 001371e8ed3e.net.in|Addresses: 10.241.65.153, 10.84.1.101 Name: 00e06f5bd42a.net.in|Addresses: 10.72.19.218,... (6 Replies)
Discussion started by: imas
6 Replies

2. Shell Programming and Scripting

Getting required fields from a test file in required fromat in unix

My data is something like shown below. date1 date2 aaa bbbb ccccc date3 date4 dddd eeeeeee ffffffffff ggggg hh I want the output like this date1date2 aaa eeeeee I serached in the forum but didn't find the exact matching solution. Please help. (7 Replies)
Discussion started by: rdhanek
7 Replies

3. Shell Programming and Scripting

How to Sort Floating Numbers Using the Sort Command?

Hi to all. I'm trying to sort this with the Unix command sort. user1:12345678:3.5:2.5:8:1:2:3 user2:12345679:4.5:3.5:8:1:3:2 user3:12345687:5.5:2.5:6:1:3:2 user4:12345670:5.5:2.5:5:3:2:1 user5:12345671:2.5:5.5:7:2:3:1 I need to get this: user3:12345687:5.5:2.5:6:1:3:2... (7 Replies)
Discussion started by: daniel.gbaena
7 Replies

4. UNIX for Dummies Questions & Answers

How to insert alternative columns and sort text from first column to second?

Hi Everybody, I am just new to UNIX as well as to this forum. I have a text file with 10,000 coloumns and each coloumn contains values separated by space. I want to separate them into new coloumns..the file is something like this as ad af 1 A as ad af 1 D ... ... 1 and A are in one... (7 Replies)
Discussion started by: Unilearn
7 Replies

5. UNIX for Advanced & Expert Users

Script to sort the files and append the extension .sort to the sorted version of the file

Hello all - I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
Discussion started by: pankaj80
3 Replies

6. Shell Programming and Scripting

Want to sort a file using awk & sed to get required output

Hi All, Need Suggestion, Want to sort a file using awk & sed to get required, output as below, such that each LUN shows correct WWPN and FA port Numbers correctly: Required output: 01FB 10000000c97843a2 8C 0 01FB 10000000c96fb279 9C 0 22AF 10000000c97843a2 8C 0 22AF 10000000c975adbd ... (10 Replies)
Discussion started by: aix_admin_007
10 Replies

7. Shell Programming and Scripting

Help with sort word and general numeric sort at the same time

Input file: 100%ABC2 3.44E-12 USA A2M%H02579 0E0 UK 100%ABC2 5.34E-8 UK 100%ABC2 3.25E-12 USA A2M%H02579 5E-45 UK Output file: 100%ABC2 3.44E-12 USA 100%ABC2 3.25E-12 USA 100%ABC2 5.34E-8 UK A2M%H02579 0E0 UK A2M%H02579 5E-45 UK Code try: sort -k1,1 -g -k2 -r input.txt... (2 Replies)
Discussion started by: perl_beginner
2 Replies

8. Shell Programming and Scripting

Sort help: How to sort collected 'file list' by date stamp :

Hi Experts, I have a filelist collected from another server , now want to sort the output using date/time stamp filed. - Filed 6, 7,8 are showing the date/time/stamp. Here is the input: #---------------------------------------------------------------------- -rw------- 1 root ... (3 Replies)
Discussion started by: rveri
3 Replies

9. UNIX for Dummies Questions & Answers

Best Alternative for checking input parameter contains required value or not

Any good way to check if code has the required output # /sbin/sysctl net.ipv4.icmp_echo_ignore_broadcasts net.ipv4.icmp_echo_ignore_broadcasts = 1 /sbin/sysctl net.ipv4.icmp_echo_ignore_broadcasts | grep "= 1" net.ipv4.icmp_echo_ignore_broadcasts = 1 What I can think of is above, and it... (16 Replies)
Discussion started by: alvinoo
16 Replies

10. Shell Programming and Scripting

How to Modify a file content in UNIX and sort for only required fields ?

I have the below contents in a file after making the below curl call curl ... | grep -E "state|Rno" | paste -sd',\n' | grep "Disconnected" > test "state" : "Disconnected",, "Rno" : "5554f1d2" "state" : "Disconnected",, "Rno" : "10587563" "state" : "Disconnected",, "Rno" :... (2 Replies)
Discussion started by: Vaibhav H
2 Replies
ici::doc::pod3::sdrhash(3)				       ICI library functions					ici::doc::pod3::sdrhash(3)

NAME
sdrhash - Simple Data Recorder hash table management functions SYNOPSIS
#include "sdr.h" Object sdr_hash_create (Sdr sdr, int keyLength, int estNbrOfEntries, int meanSearchLength); int sdr_hash_insert (Sdr sdr, Object hash, char *key, Address value, Object *entry); int sdr_hash_delete_entry (Sdr sdr, Object entry); int sdr_hash_entry_value (Sdr sdr, Object hash, Object entry); int sdr_hash_retrieve (Sdr sdr, Object hash, char *key, Address *value, Object *entry); int sdr_hash_count (Sdr sdr, Object hash); int sdr_hash_revise (Sdr sdr, Object hash, char *key, Address value); int sdr_hash_remove (Sdr sdr, Object hash, char *key, Address *value); int sdr_hash_destroy (Sdr sdr, Object hash); DESCRIPTION
The SDR hash functions manage hash table objects in an SDR. Hash tables associate values with keys. A value is always in the form of an SDR Address, nominally the address of some stored object identified by the associated key, but the actual significance of a value may be anything that fits into a long. A key is always an array of from 1 to 255 bytes, which may have any semantics at all. Keys must be unique; no two distinct entries in an SDR hash table may have the same key. Any attempt to insert a duplicate entry in an SDR hash table will be rejected. All keys must be of the same length, and that length must be declared at the time the hash table is created. Invoking a hash table function with a key that is shorter than the declared length will have unpredictable results. An SDR hash table is an array of linked lists. The location of a given value in the hash table is automatically determined by computing a "hash" of the key, dividing the hash by the number of linked lists in the array, using the remainder as an index to the corresponding linked list, and then sequentially searching through the list entries until the entry with the matching key is found. The number of linked lists in the array is automatically computed at the time the hash table is created, based on the estimated maximum number of entries you expect to store in the table and the mean linked list length (i.e., mean search time) you prefer. Increasing the maximum number of entries in the table and decreasing the mean linked list length both tend to increase the amount of SDR heap space occupied by the hash table. Object sdr_hash_create(Sdr sdr, int keyLength, int estNbrOfEntries, int meanSearchLength) Creates an SDR hash table. Returns the SDR address of the new hash table on success, zero on any error. int sdr_hash_insert(Sdr sdr, Object hash, char *key, Address value, Object *entry) Inserts an entry into the hash table identified by hash. On success, places the address of the new hash table entry in entry and returns zero. Returns -1 on any error. int sdr_hash_delete_entry(Sdr sdr, Object entry) Deletes the hash table entry identified by entry. Returns zero on success, -1 on any error. Address sdr_hash_entry_value(Sdr sdr, Object hash, Object entry) Returns the value of the hash table entry identified by entry. int sdr_hash_retrieve(Sdr sdr, Object hash, char *key, Address *value, Object *entry) Searches for the value associated with key in this hash table, storing it in value if found. If the entry matching key was found, places the address of the hash table entry in entry and returns 1. Returns zero if no such entry exists, -1 on any other failure. int sdr_hash_count(Sdr sdr, Object hash) Returns the number of entries in the hash table identified by hash. int sdr_hash_revise(Sdr sdr, Object hash, char *key, Address value) Searches for the hash table entry matching key in this hash table, replacing the associated value with value if found. Returns 1 if the entry matching key was found, zero if no such entry exists, -1 on any other failure. int sdr_hash_remove(Sdr sdr, Object hash, char *key, Address *value) Searches for the hash table entry matching key in this hash table; if the entry is found, stores its value in value, deletes the entry, and returns 1. Returns zero if no such entry exists, -1 on any other failure. void sdr_hash_destroy(Sdr sdr, Object hash); Destroys hash, destroying all entries in all linked lists of the array and destroying the hash table array structure itself. DO NOT use sdr_free() to destroy a hash table, as this would leave the hash table's content allocated yet unreferenced. SEE ALSO
sdr(3), sdrlist(3), sdrtable(3) perl v5.14.2 2012-05-25 ici::doc::pod3::sdrhash(3)
All times are GMT -4. The time now is 10:49 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy