Programming

If x*y > SIZE_MAX, because of that overflow, malloc will silently attempt to allocate an amount of storage less than what was intended. This won't happen with a properly implemented calloc(x, y). If the overhead of zeroing the allocation is undesirable, then explicitly checking for overflow before using malloc would be wise.


I have, but that was a while ago. These days, the case of the absent header, that you mentioned, is a much more likely scenario.

Regards,
Alister

If you're allocating enough memory to approach SIZE_MAX, you really, really don't want to zero it first

Please bring in 2 GB worth of pages kernel; I'll wait, lol.

It's not much of a wait. On my 5 yr old laptop, memset()-ing 2 GB takes a fraction of a second. I didn't time the memset() itself, but it took less than one quarter of a second to complete a fork-exec-main-malloc-memset-exit sequence, which determines the upper bound with plenty of padding.

Even so, memset() is largely irrelevant in this scenario. For a single, unusually large allocation, you're practically guaranteed that there won't be a suitable chunk already available within the allocator's store. In that case, the requested pages will need to be allocated with sbrk or mmap and those pages will be zeroed by the kernel regardless. I repeat, whether you call malloc or calloc, in this scenario, you will be getting zeroed memory. And, with calloc(), you get complimentary overflow checking. WOOHOO!

I looked over the glibc, FreeBSD, and OpenBSD calloc() implementations and they never re-zero memory that's freshly delivered by sbrk() or mmap(). Should anyone care to have a look for themselves, for your convenience:
glibc malloc.c
FreeBSD malloc.c
OpenBSD malloc.c

Furthermore, it's quite possible that alll of those zeroed pages are backed by a single zeroed page that the kernel keeps around specifically for such situations. In that case, a real page won't be mapped until something is written.

A last point. If you really, really, really truly care about squeezing every last ounce of performance from the allocators (some folks do even when it's unwarranted), then don't use malloc and friends. They're designed to be general purpose routines and the tradeoffs involved beget complexity. A stupid, simple, single-purpose, tailor-made solution may be called for (although this is very seldom the case).

Regards,
Alister

Imagine how many times a second you could do that if you weren't memsetting.

Besides, you're confused between 'allocate' and 'page in'. Blanking two gigs of RAM at once means paging in two gigs of RAM, turfing two gigs of perfectly cache, etc, etc. That can have major performance consequences for everything else. Burning a CD when something pages out all your cache for instance can produce an instant coaster. Double the work for no gain is never irrelevant. Yes -- as needed. NOT all at once. Only when paged in. I repeat. There is a difference between allocating memory and paging it in.

---------- Post updated at 10:19 AM ---------- Previous update was at 09:55 AM ----------

This is a good find

With all due respect, Corona688, you're the one that's confused.

I am aware that allocation and paging in are discreet steps. What I wrote above in no way suggests that memset() does not incur further overhead beyond sbrk or malloc. Quite the contrary, I ran that memset() to demonstrate that even with the further overhead and work, zeroing 2 GB of ram does not take very long.

As for the rest of my post, the gist is that there is no difference between calloc and malloc for a large allocation whose pages aren't already mapped in.

Why not? If the memory needs to be zeroed, it needs to be zeroed. The size of the allocation is irrelevant. For a small allocation using a recycled chunk, the memset hit is negligible. For the huge allocations discussed, if there is sufficient memory for the allocation to succeed, calloc() knows that the allocation will be backed by fresh pages that are already zeroed. Knowing this, calloc() will not call memset(), and so no paging in will occur until the memory is written to. In the end, calloc() and malloc() are equivalent.

Regards,
Alister

if you read my second post below my first one, you'll see I realized the implications.

Whether my CPU and memory bus have cycles to waste really isn't related to whether certain functions waste them, which did a good job of confusing me to your point. I still don't understand what your benchmarks are supposed to prove. Luckily it doesn't matter.