Programming

Agreed... Though at 60% utilization you are also getting worryingly close to 32-bit address size limits. The heap can't use all 4 gigs because there's other things in the way.

Ok, more accurately but still quite roughly:
at the starting of the C++ program the OS is using ~0.3GB or ram. The program stores ~3000 text files of unknown size in ram, one by one, via realloc(). All the files should fit into less than 2GB and, anyway, when realloc() fails there are still ~0.8GB available (40% of 2GB). At that time realloc() has allocated 2-0.3-0.8 ~=0.9GB.
Each file is stored independently: realloc() makes space for one single file at each time. The space is allocated in increment of 500KB. It means that if a file is 800KB, realloc() works twice and allocates 1MB.

It happens that when the allocated memory reaches ~1.2GB (OS=0.3, program=0.9), realloc() returns NULL when attempting to allocated another 500KB.

As long as realloc() does not allocated more than those ~0.9GB, everything works just fine.

Thanks again.

You are focusing too much on RAM. realloc() doesn't care or even use RAM. What this function does is reserving (or releasing) virtual memory. Whether this virtual memory is backed by RAM or not doesn't matter.

You should monitor your process virtual memory usage and see if you don't exhaust it. You might also see how scattered your process virtual memory is with using "pmap -d pid".

And by the way, why aren't you directly allocating the file size ? or even better, why aren't you using mmap to directly load the files in memory ?

Thanks! Will do and eventually will post again.

---------- Post updated at 04:52 PM ---------- Previous update was at 04:34 PM ----------

Because I do NOT know the size of the files. They are coming from the Internet. Instead of storing them first on HD and then loading them in memory, it's *much* faster receiving them directly in memory.

---------- Post updated at 06:25 PM ---------- Previous update was at 04:52 PM ----------

Ok, while I'm digging into something that I never faced (memory management), I'd like to take a short cut and ask you guys how I should interpret the output of vmstat and pmap.

In quiet mode, before launching the infamous C++ program here is the output of vmstat:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 900908 64892 727084 0 0 7 14 154 62 4 1 95 0

Then I launch the program.

In the middle of its exec time `pmap -x PID | grep anon` gives (after grep'ing out spaces smaller than 1000):
59e17000 5904 - - - rw--- [ anon ]
5a3db000 1468 - - - rw--- [ anon ]
5a54b000 1468 - - - rw--- [ anon ]
5a6bb000 63468 - - - rw--- [ anon ]
5e6a0000 24108 - - - rw--- [ anon ]
60015000 6396 - - - rw--- [ anon ]
6083e000 189420 - - - rw--- [ anon ]
6c323000 165804 - - - rw--- [ anon ]
766f8000 39852 - - - rw--- [ anon ]
78de3000 1468 - - - rw--- [ anon ]
78f52000 16236 - - - rw--- [ anon ]
7a117000 119064 - - - rw--- [ anon ]
81652000 122016 - - - rw--- [ anon ]
88e6f000 34932 - - - rw--- [ anon ]
8b3e6000 3444 - - - rw--- [ anon ]
8b92d000 202704 - - - rw--- [ anon ]
97f9b000 94956 - - - rw--- [ anon ]
9dfb0000 69864 - - - rw--- [ anon ]
a25d4000 135300 - - - rw--- [ anon ]
aaaea000 114636 - - - rw--- [ anon ]
b1add000 1468 - - - rw--- [ anon ]
b1c4d000 1468 - - - rw--- [ anon ]
b1dbd000 91788 - - - rw--- [ anon ]
total kB 1541848

Just before realloc() fails here is again `pmap -x PID | grep anon` (after grep'ing out spaces smaller than 1000):

00adc000 1476 - - - rw--- [ anon ]
00ee4000 116112 - - - rw--- [ anon ]
080c9000 25708 - - - rw--- [ anon ]
099f1000 198768 - - - rw--- [ anon ]
15c0d000 1468 - - - rw--- [ anon ]
15d7d000 1468 - - - rw--- [ anon ]
15eed000 91020 - - - rw--- [ anon ]
1b846000 352764 - - - rw--- [ anon ]
310c5000 1956 - - - rw--- [ anon ]
312ae000 1468 - - - rw--- [ anon ]
3141e000 61992 - - - rw--- [ anon ]
35292000 5412 - - - rw--- [ anon ]
358d0000 31488 - - - rw--- [ anon ]
3797a000 49692 - - - rw--- [ anon ]
3aa78000 137268 - - - rw--- [ anon ]
4326f000 7872 - - - rw--- [ anon ]
43c09000 28536 - - - rw--- [ anon ]
459d1000 20172 - - - rw--- [ anon ]
46e79000 39360 - - - rw--- [ anon ]
495de000 34440 - - - rw--- [ anon ]
4b96a000 7872 - - - rw--- [ anon ]
4c304000 10824 - - - rw--- [ anon ]
4cf80000 208116 - - - rw--- [ anon ]
59b35000 8856 - - - rw--- [ anon ]
5a3db000 1468 - - - rw--- [ anon ]
5a54b000 1468 - - - rw--- [ anon ]
5a6bb000 63468 - - - rw--- [ anon ]
5e6a0000 24108 - - - rw--- [ anon ]
60015000 6396 - - - rw--- [ anon ]
6083e000 189420 - - - rw--- [ anon ]
6c323000 165804 - - - rw--- [ anon ]
766f8000 39852 - - - rw--- [ anon ]
78e5c000 17220 - - - rw--- [ anon ]
7a117000 119064 - - - rw--- [ anon ]
81652000 122016 - - - rw--- [ anon ]
88e6f000 34932 - - - rw--- [ anon ]
8b3e6000 3444 - - - rw--- [ anon ]
8b92d000 202704 - - - rw--- [ anon ]
97f9b000 94956 - - - rw--- [ anon ]
9dfb0000 69864 - - - rw--- [ anon ]
a25d4000 135300 - - - rw--- [ anon ]
aaaea000 114636 - - - rw--- [ anon ]
b1add000 1468 - - - rw--- [ anon ]
b1c4d000 1468 - - - rw--- [ anon ]
b1dbd000 91788 - - - rw--- [ anon ]
b7772000 65936 - - - rw--- [ anon ]
bb7d6000 3420 - - - rw--- [ anon ]
bbb2d000 43296 - - - rw--- [ anon ]
total kB 3112744 - - -

How should I interpret these info?


---------- Post updated at 06:29 PM ---------- Previous update was at 06:25 PM ----------

Mistakenly, I posted some further info as a reply to myself. Can you guys pls take a look at my last post?

Thanks!

Why would that be? All memory is swap backed technically, this is just disk-backed instead. If you save it on disk it'll be in cache, and cache is memory too. It gets paged in on-demand just like any other memory, with the advantage of not wasting swap space. It's quite efficient actually.

---------- Post updated at 12:08 AM ---------- Previous update was at 12:06 AM ----------

That you've used up 3 gigabytes of address space if not memory, way beyond the limit I'd have suspected your program would begin to bug out on. In any case the reason for the errors is clear. You're running out of address space.

mmap() might be able to fill in the corners you're not using, but there's not a lot left to be had.

Yes. Either buy some ram and move to 64 bit or fix your application design. The OS is better suited to store thousands of files than a user application.
If you feel 3+ GB of memory isn't what your application should use, maybe is there a memory leak in it. You might use valgrind to investigate that.

When you're reading from a network, writing to disk is going to be much, much faster than your network speeds unless your disks are stone tablets with data being carved into them by gnomes.

And what do you plan on doing WHEN you run into a bug and your program crashes in production with all these files in RAM? Did you even think of that?