Programming

Hi all

I'm trying to use someone else's software, which has a realloc that fails in it. This is probably due to memory limitations, as it only happens when I use this software on huge datasets.

First question : how to diagnose if it's a soft or hard limitation? I mean, if it's due to my computer's memory limits, or to some limits the compiler / my C implementation imposes?
I'm running Mandriva Linux, 64 bit, with 4Gb RAM and 6GB swap, I don't think I'm reaching those limits.
The same failure happens on our computer cluster, which has about 60Gb of RAM ...

I used the makefile that came with the software, which adds these options to the gcc command :
CFLAGS = -O4 -funroll-loops -fomit-frame-pointer -finline-functions

More questions might follow depending on the answer, but the obvious question is : how to get around this limit ?

Well, you can check that the system isn't artificially imposing a limit on you by running `ulimit -m' which will tell you the maximum number of kB of memory the OS will allocate to you. If that reports anything other than "unlimited" you can increase the limit with `ulimit -m unlimited', but you need to be root to do this.

Other than that, you could always get realloc() to print the size of the memory it failed to move, to make sure you're not trying to move 2 GB of data or something ridiculous.

Code:

[jos@faba ~]$ ulimit -m
unlimited

But you're right, the program is trying to do something ridiculous :

Code:

   k = com.npatt*com.ncatG*sizeof(double);
   if((com.fhK=(double*)realloc(com.fhK,k)) == NULL) 
    {
    printf("%d\n",k);
    error2("oom fhK");
}

Upon execution, the reported value for k is -169522176.

Thanks a lot for your suggestion, I hadn't thought about that !

jos

As an aside, whoever wrote the makefile was trying to be really aggressive with the optimizations. Perhaps they were using bad algorithms to start with, and thought to speed things up by cranking up the compiler optimizations. At any rate, these optimizing switches sometimes do, but sometimes don't, speed up execution time. Only a careful analysis (profiling) can provide the answer. At any rate, GCC offers up to -O3. -O4 is not an option, to my knowledge.

Thanks for pointing that out, kermit !
I'll keep that in mind for the next time I analyze large datasets.

jos

The omit-frame-pointer and inline-functions are enabled by default if using -O3 optimization - so this is all that is required

Code:

CFLAGS = -O3 -funroll-loops

...and funroll-loops isn't often helpful.

If you're still debugging you should have -O0 and -ggdb. No optimizations and debugging support.