10-23-2012
I believe that 'sort -u' saves just the first occurrance of the unique key, so you sort first non-unique to get the right first record saved.
However, I agree it is a bit of a shame to use so much storage and processing when you could just tuck them in an associative array and overwrite any old values, especially in cases where the data starts out sorted in some relevant way. The only drawback is that the speed of shell operations might be a drag on big volume. You can scale up sort in parallel using pipes and sort -m, but for the unsorted lookup solution at machine speed, C++ or at least JAVA can work a hash table faster, and you can pre-size the hash table big enough to get good use of RAM and VM in even a 32 bit app. I like big powers of 2, since a modulus of the hash becomes a lower bit mask. Empty hash table entries are just pointers in an array, 4 or 8 bytes cost each, which is pretty cheap, and does no harm for smaller data sets! Hash beats tree for query and churn speed, but tree does provide sorted output and scales more automatically. Linear hash (tables in power of two sizes that can double the hash table size for congested buckets) has a better dynamic scaling, but slower query and churn than straight hash. I have not found a lot of hash implementations that reveal they are linear.
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Hi All,
I have a file 1.txt which has the duplicate dns entries as shown:
Name: 000f9fbc6738.net.in|Addresses: 10.241.66.169, 10.84.2.222,212.241.66.170
Name: 001371e8ed3e.net.in|Addresses: 10.241.65.153, 10.84.1.101
Name: 00e06f5bd42a.net.in|Addresses: 10.72.19.218,... (6 Replies)
Discussion started by: imas
6 Replies
2. Shell Programming and Scripting
My data is something like shown below.
date1 date2 aaa bbbb ccccc
date3 date4 dddd eeeeeee ffffffffff ggggg hh
I want the output like this
date1date2 aaa eeeeee
I serached in the forum but didn't find the exact matching solution. Please help. (7 Replies)
Discussion started by: rdhanek
7 Replies
3. Shell Programming and Scripting
Hi to all.
I'm trying to sort this with the Unix command sort.
user1:12345678:3.5:2.5:8:1:2:3
user2:12345679:4.5:3.5:8:1:3:2
user3:12345687:5.5:2.5:6:1:3:2
user4:12345670:5.5:2.5:5:3:2:1
user5:12345671:2.5:5.5:7:2:3:1
I need to get this:
user3:12345687:5.5:2.5:6:1:3:2... (7 Replies)
Discussion started by: daniel.gbaena
7 Replies
4. UNIX for Dummies Questions & Answers
Hi Everybody,
I am just new to UNIX as well as to this forum. I have a text file with 10,000 coloumns and each coloumn contains values separated by space. I want to separate them into new coloumns..the file is something like this
as ad af 1 A
as ad af 1 D
...
...
1 and A are in one... (7 Replies)
Discussion started by: Unilearn
7 Replies
5. UNIX for Advanced & Expert Users
Hello all -
I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
Discussion started by: pankaj80
3 Replies
6. Shell Programming and Scripting
Hi All, Need Suggestion, Want to sort a file using awk & sed to get required, output as below, such that each LUN shows correct WWPN and FA port Numbers correctly:
Required output:
01FB 10000000c97843a2 8C 0
01FB 10000000c96fb279 9C 0
22AF 10000000c97843a2 8C 0
22AF 10000000c975adbd ... (10 Replies)
Discussion started by: aix_admin_007
10 Replies
7. Shell Programming and Scripting
Input file:
100%ABC2 3.44E-12 USA
A2M%H02579 0E0 UK
100%ABC2 5.34E-8 UK
100%ABC2 3.25E-12 USA
A2M%H02579 5E-45 UK
Output file:
100%ABC2 3.44E-12 USA
100%ABC2 3.25E-12 USA
100%ABC2 5.34E-8 UK
A2M%H02579 0E0 UK
A2M%H02579 5E-45 UK
Code try:
sort -k1,1 -g -k2 -r input.txt... (2 Replies)
Discussion started by: perl_beginner
2 Replies
8. Shell Programming and Scripting
Hi Experts,
I have a filelist collected from another server , now want to sort the output using date/time stamp filed.
- Filed 6, 7,8 are showing the date/time/stamp.
Here is the input:
#----------------------------------------------------------------------
-rw------- 1 root ... (3 Replies)
Discussion started by: rveri
3 Replies
9. UNIX for Dummies Questions & Answers
Any good way to check if code has the required output
# /sbin/sysctl net.ipv4.icmp_echo_ignore_broadcasts
net.ipv4.icmp_echo_ignore_broadcasts = 1
/sbin/sysctl net.ipv4.icmp_echo_ignore_broadcasts | grep "= 1"
net.ipv4.icmp_echo_ignore_broadcasts = 1
What I can think of is above, and it... (16 Replies)
Discussion started by: alvinoo
16 Replies
10. Shell Programming and Scripting
I have the below contents in a file after making the below curl call
curl ... | grep -E "state|Rno" | paste -sd',\n' | grep "Disconnected" > test
"state" : "Disconnected",, "Rno" : "5554f1d2"
"state" : "Disconnected",, "Rno" : "10587563"
"state" : "Disconnected",, "Rno" :... (2 Replies)
Discussion started by: Vaibhav H
2 Replies
cdb(5) File Formats Manual cdb(5)
NAME
cdb - Constant DataBase file format
DESCRIPTION
A cdb database is a single file used to map `keys' to `values', having records of (key,value) pairs. File consists of 3 parts: toc (table
of contents), data and index (hash tables).
Toc has fixed length of 2048 bytes, containing 256 pointers to hash tables inside index sections. Every pointer consists of position of a
hash table in bytes from the beginning of a file, and a size of a hash table in entries, both are 4-bytes (32 bits) unsigned integers in
little-endian form. Hash table length may have zero length, meaning that corresponding hash table is empty.
Right after toc section, data section follows without any alingment. It consists of series of records, each is a key length, value (data)
length, key and value. Again, key and value length are 4-byte unsigned integers. Each next record follows previous without any special
alignment.
After data section, index (hash tables) section follows. It should be looked to in conjunction with toc section, where each of max 256
hash tables are defined. Index section consists of series of hash tables, with starting position and length defined in toc section. Every
hash table is a sequence of records each holds two numbers: key's hash value and record position inside data section (bytes from the begin-
ning of a file to first byte of key length starting data record). If record position is zero, then this is an empty hash table slot,
pointed to nowhere.
CDB hash function is
hv = ((hv << 5) + hv) ^ c
for every single c byte of a key, starting with hv = 5381.
Toc section indexed by (hv % 256), i.e. hash value modulo 256 (number of entries in toc section).
In order to find a record, one should: first, compute the hash value (hv) of a key. Second, look to hash table number hv modulo 256. If
it is empty, then there is no such key exists. If it is not empty, then third, loop by slots inside that hash table, starting from slot
with number hv divided by 256 modulo length of that table, or ((hv / 256) % htlen), searching for this hv in hash table. Stop search on
empty slot (if record position is zero) or when all slots was probed (note cyclic search, jumping from end to beginning of a table). When
hash value in question is found in hash table, look to key of corresponding record, comparing it with key in question. If them of the same
length and equals to each other, then record is found, overwise, repeat with next hash table slot. Note that there may be several records
with the same key.
SEE ALSO
cdb(1), cdb(3).
AUTHOR
The tinycdb package written by Michael Tokarev <mjt@corpit.ru>, based on ideas and shares file format with original cdb library by Dan
Bernstein.
LICENSE
Public domain.
Apr, 2005 cdb(5)