UNIX for Beginners Questions & Answers

Don
Thanks! Is there any way to sort the data directly using awk instead of "handing" the task to sort?

Not with standard awk. GNU awk (sometimes invoked as gawk) includes two built-in functions (asort() and asorti()) that can be used to sort arrays. I'm guessing that asorti() could be used to sort an array whose indices are your record's 7th colon separated field values, but these functions are not available in the awk on my system (so I have no way of testing it out). And, of course, you could write your own function in awk to sort an array. Any of these array sorting functions would require that you load your entire input file (other than skipping record #1) into an array in memory, sorting the array, and then writing the sorted array. Your current awk script never keeps more than one input record in memory. (Note that the sort utility uses temporary files when necessary if the file(s) being sorted are too large to fit in memory. You haven't said how large your input file is (in bytes) although you have said it can have up to 15 million records, so I don't know if this would be a concern trying to use awk instead of sort to sort your data on your system.)

Don
My files are ~8,000,000kb. Would that limit my ability to sort the records using awk instead of sort?

Hi Xterra,
Do you have enough memory on your system for awk to build an array larger than 8 terabytes?

I will give it a try on my cluster

It is, ahem, highly unlikely anybody can peruse something with 8TB of RAM.

If your input file is relatively static (like there will only be lines appended but rarely to never lines get deleted) you might try to create a smaller file with just your sort-key and a line-number. This file should be considerably smaller (maybe several hundred MB or a few GB) and the might be possible to sort in memory.

You would have to go through this file and either rewrite your big file or filter out the subset you are interested in using the line numbers, which will perhaps take a long time again, but if the file changes not that often (see above) you will have to redo only parts of it, so this might help anyway.

Along the same lines: wouldn't a database with an indexed table be what you want? Databases have methods to deal with files that are bigger as the available main memory. So what you are doing here is perhaps old news for DB-software

I hope this helps.

bakunin

Wouldn't 8.000.000 kB (8 * 10^6 * 10^3) be 8 GB? And thus (sort of) manageable? How come we're talking terabytes?