Quote:
Originally Posted by
DreamWarrior
I'm not convinced on the fragmentation "issue". It may speed things up because you call malloc fewer times, but...
A bit of fragmentation doesn't hurt malloc() much. A small percentage of wasted space? Boo hoo. Unless it's truly horrific you'd never notice or care.
The cost of fragmentation for repeated
realloc() is exponentially-growing wasted CPU time as your existing data leapfrogs through every available heap hole in order of size. I think that's worth minimizing.
If you knew the sizes of your heap pools, you could aim large enough to probably be at the end of a pool but small enough to not spill into the next one. We don't, but exponential growth is a decent approximation.
To a point, anyway. For ludicrous sizes I'll admit a cap might be good.
Quote:
Either way...it's a blanket statement that one is better than the other. If you had a process that was multi-threaded and read ten million items in with ten threads would you ask for it exponentially in each thread?
I think that kind of situation needs careful tuning no matter what allocation method you pick.
I might use large fixed-size blocks of memory anonymously mmap()-ed in, one to a thread. That'd let you parcel out fractions of address space pretty precisely without exhausting your memory map or wasting memory. If you knew a little about your address space you could pick an entire contiguous region for them and MAP_FIXED things into place to prevent your address space from growing motheaten.
You could even extend that to accommodate more items than you can hold in your address space by making the mappings file-backed... if you fill a mapping, just truncate() the file one chunk larger and move your map further into the file. This'd also prevent you from exhausting system memory and swap. To sort it, you could qsort in chunks then merge the chunks.