Shell Programming and Scripting

Hello

If you wanted to sort a .csv file that was filled with lines like this:

<Ticker>,<Date as YYYYMMDD>,<Time as H:M:S>,<Volume>,<Corr>

(H : [1, 23], M, S: [0, 59])

by date, does anybody know of a better solution than to turn the 3rd and 4th colons of every line into commas, sorting on four keys, and then turning those two commas in every line back to colons? It seems very inefficient to me. (I would just do it and not bother asking if these files weren't 50+GB.)

---------- Post updated at 09:43 PM ---------- Previous update was at 09:27 PM ----------

Meh, I'll let it run overnight.

Code:

sed 's/:/,/g' big_file.csv | sort -k 2,2 -k 3,3 -k 4,4 -k 5,5 -t',' | sed 's/,/:/3' | sed 's/,/:/3' > big_file.sorted.csv

how about:

Code:

sort -t , -k2n

Try this one

Code:

sed 's/:/,/g' test_sort| sort -t ',' -k 2,7 |sed 's/,/:/3'|sed 's/,/:/3'> test_org_final| mv test_org_final test_sort

Sort broke @ 2am:

Code:

read failed: /tmp/sortOgLpWg: Input/ouput error

This is a real problem now. Does anybody know of a way to sort a humongous file that won't (likely) break?

Are hours, minutes and seconds all zero padded? For example, 01:02:03 instead of 1:2:3 or 1:02:03? If so, you do not need to modify anything. You can use the default lexicographical sort with the date and time fields as the keys.

Also, you mentioned that hours range betwee 1-23. In case it's relevant, that's only a 23 hour day.

If the source file is 50+ GB, you are going to need a lot of ram. You'll probably need to split the file into smaller chunks, sort them individually, and then merge them with sort -m.

Regards,
Alister

Oddly the hours aren't zero padded but the minutes and seconds are. (I think it's like [1]?[0-9]:[0-5][0-9]:[0-5][0-9] in Regex-speak.)

I'm going to try to figure out how to split it up and then attempt sorting again -- thanks.

By the way, the consecutive seds in the pipeline can be simplified: sed 's/,/:/3 ; s/,/:/3'.

That'll save some time in context switches and copying data in and out of kernel/userland buffers.

Regards,
Alister

---------- Post updated at 02:55 PM ---------- Previous update was at 02:53 PM ----------

Also, it seems GNU sort can handle this situation, by automatically creating tmp files during the sorting process. I'm assuming you're not on Linux. If so, and if you are using GNU sort, you should paste the exact error message.