Shell Programming and Scripting

I'm using Ubuntu Jaunty.

But unless somebody can come up with a solution within the next hour I'm just going to write something to manually split and sort it.

I was just looking at a few sort implementations, they all already use TEMPDIR to store sorted chunks when dealing with very large input. You can try doing it manually, but your sort tool is probably already doing that for you.

Good luck.

Context switching isnt limited just to the pipeline processes...the fewer the processes on the system the lesser the context switching...hope you catch my drift.
The sort on my aix and hpux boxes is able to sort the entire h:m:s field not just upto the first colon...perhaps your sort has a limitation mine doesnt.

You misunderstand how sort works.

The moment that sort sees that colon in the numeric key, it stops evaluating the key. It will not go any further. All times with the same hour will compare equal because the minutes and seconds are not looked at. At this point, to decide how to order all of those equal records, sort will then look at the rest of the line ... starting with the first character. Also, it will not do so numerically, but lexographically.

The following two excerpts are from the POSIX sort standard, Man Page for sort (POSIX Section 1) - The UNIX and Linux Forums

Unless the colon is a radix or thousands separator character in the current locale (which is not the case in any locale that I'm aware of), it's not a valid part of the numeric string. The numeric key comparison will end right there.



Looking again at what you proposed:
We can see that the third field numeric key will not be inspected beyond the hours portion. All times with the same hour will compare equal regardless of the value of minutes and seconds. To then decide how to order the lines which have compared equal, the first field will be considered and then everything from the first colon in the third field inclusive till the end of the line. This is obviously wrong.

If your sort(1) does not behave as described above, it's broken (with regard to posix compliancy).


Here's a sample to illustrate the point:

Code:

$ cat data
b,11:00:59
a,11:45:00
c,11:15:00
d,11:01:00
e,11:00:00

The following does not give the desired result because the numerical comparison ends at the first colon of the second field. Therefore, every single line compares as equal (key evaluates to 11). At that point, sort then looks at the entire line. Note how in this example the output order is essentially a sort on the first field.

Code:

$ sort -t, -k2,2n data
a,11:45:00
b,11:00:59
c,11:15:00
d,11:01:00
e,11:00:00

To get a sort on the time using numeric comparison (which is not really necessary in this case since each time component is two digits), you need to work around the colon with finer grained keys. The following almost gets us there, but note that in this case, the seconds are not considered and records with the same hour and minute are sorted according to the first field (the first two lines are incorrect):

Code:

$ sort -t, -k2.1,2.2n -k2.4,2.5n data
b,11:00:59
e,11:00:00
d,11:01:00
c,11:15:00
a,11:45:00

The correct solutions for this example:

Code:

$ sort -t, -k2.1,2.2n -k2.4,2.5n -k2.7,2.8n data
e,11:00:00
b,11:00:59
d,11:01:00
c,11:15:00
a,11:45:00
$ sort -t, -k2,2 data
e,11:00:00
b,11:00:59
d,11:01:00
c,11:15:00
a,11:45:00

Although neither of these solutions is valid for this thread's problem since they both depend on the time having a fixed format.

Regards,
Alister

Thanks for all your help and insight, alister.

I just thought I would come on here and explain what I ended up doing.

Problem: Server has too little hard-drive space to perform sort, but plenty of RAM, whereas local machine has plenty of hard-drive space to perform sort, but not enough RAM.

Solution:

Code:

-- Remote Server --

$ cat data/run.sh

# Location with enough space to hold one copy of file
tmp_dir="/devlinux_work3/<name>/tmp" 

sed 's/:/,/g' $1 | sort -k 2,2 -k 3,3 -k 4,4 -k 5,5 -t',' -T ${tmp_dir} | sed 's/,/:/3 ; s/,/:/3'

-- Local Machine --

plink <name>@<machine>.<company> "bash /devlinux_work4/<name>/data/run.sh /devlinux_work4/<name>/data/trades.csv" > trades_sorted.csv

You're quite welcome. And thank you for reporting back.

Regards,
Alister

Your sort option may not be correct (-k 3,3n) since you said the hour part of the date is not zero padded.
Your server machine can hold one copy of the data, that's enough to do the sort there. You can compress the input file and/or pipe the sort to another compress to do the same with the output file. Don't worry about extra pipe lines, they are every efficient (especially on multi-core servers). Dealing with compressed files sometimes is more efficient due to much less I/O although very CPU intensive. Keep in mind that in most cases you can avoid un-compressing the files to physical files in order to process them by using pipes or named pipes.