Solaris

Hi Guys,

My client (dear clients, I hate to love you) has the funky idea of sorting a 5G flat file.

Certainly enough, this is taking forever and also fulls the / of our machine.

Any idea of how we could proceed to make this a little bit more efficient?

Maybe by forcing sort to "stay in memory" instead of writing to disk?

Any ideas?

I'm kinda clueless for the moment ...

THANKS!

"sort" is certainly able to sort a 5-gigabyte flat file. It'll take a while but it has no hard limits except disk space for temporary files(as you've discovered!)

You can tell it to use more memory with -S. It expects a number in kilobytes. 5000 would be about 5 megabytes, for example. Don't tell it to use more RAM than you have free (or cached; cache counts as free), because that will actually make it slower once the system runs out of memory and begins swapping. If you're on a 32-bit system, you should only tell it to use a gigabyte or two at the very most, because of 32-bit address size limitations.

Code:

sort -S 5000000 file_to_sort > file_output

You can prevent it from filling up / by making it store temporary files somewhere else. If you give it a folder on a different disk than the file being sorted, this may increase performance, too, by having more bandwidth available on two disks than one.

Code:

sort -T /path/to/tempdir file_to_sort > file_output

All of these flags can be found in man sort

Hi Corona,

Thanks for your answer. Before posting I always read the mans, I ask quesitons when I'm looking for a different way to make things and to try to understand a little bit the mechanisms behind some of the bins.

By looking at the process while it was doing the sort I noticed that it divides the file in multiple chunks (11megs in my case) then makes a new file out of the 64 first "chunks" it created and delete those first smaller "chunks". This will then generate 9 files (approximately) then generate a finale one from those last 9 files. In other words, we need approximately 1.50 to 1.75 x File_size of disk space to be able to complete that sort.

Because our swap device is a disk, I don't think it will actually slow down the process if I tell it to do something like :

Code:

sort -S 80% file > destination

So it'll be as fast as just writing to the disk like it is currently doing. Yes, it'll slow other processes but we don't really care at the moment because this job is necessary for any other job to start.

I realize that the question I asked is not really the question I wanted to ask. By asking a more efficient way, I was more thinking "a fastest way".

Any idea on how to "split" or "fork" sort on multiple cores? We have 8 available on the oldish v490 and I was hoping to use maybe 4 to 6 cores to do it.

If not, I guess that redirecting the sort to our SAN storage will be the most efficient way.

Thanks again Corona for your time, really appreciated.

That your swap's a separate disk means it's not as bad when it swaps, but it's still pretty bad. You do that by splitting the file into parts to sort separately, then merging to create the final result.

Have you considered loading this data into a DB, and using SQL to select the records in a sorted way, then streaming it out to a file with identical structure as your original file?

If he's running out of disk space when sorting, I doubt he'll have space for data + metadata + index.

Sure it's a valid point that if disk space max is being reached during sort, the same issue may be encountered while inserting into a DB. However, we don't really know all the details of the server. Who knows; maybe there is some additional mounted volume, or some external DB available (not on the same disk) that could be used.