Programming

Hi , experts.

I work on Linux station (RedHat 5.7), regular user, but have root password.

%> uname -a
Linux ran1log06 2.6.18-238.1.1.el5 #1 SMP Tue Jan 4 13:32:19 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
%> cat /etc/issue
Red Hat Enterprise Linux Client release 5.7 (Tikanga)
Kernel \r on an \m

My C code application allocates memory by a lot of calloc(...)

The application fails - calloc returns NULL pointer and error "Cannot allocate memory"

But I see a lot of free memory here:

%> free -t
total used free shared buffers cached
Mem: 8017116 7872188 144928 0 113356 2409372
-/+ buffers/cache: 5349460 2667656
Swap: 2097144 184 2096960
Total: 10114260 7872372 2241888


there is no problem with limit also:
%> limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize unlimited
coredumpsize unlimited
memoryuse unlimited
vmemoryuse unlimited
descriptors 1024
memorylocked 32 kbytes
maxproc 65535

I also ran it under
It runs fine in other linux, also Red Hat 5.7

What may be the problem here?
May be some limit on the process on ran1log06 machine?
How can I check it?













----------------------------------------
Note: This post may be relates to "Programming" forum.

You can fragment memory by making a lots of calls to malloc or calloc, freeing some blocks and keeping others. After a while substantially larger allocation requests fail because there is not enough free contiguous memory.

Based on your free -t output I would say you have a lot of memory. So, my best guess is fragmentation.

As a possible fix, consider adding more (way more) swap, say 50GB.

But - whatever your code does it would appear not to play well with others. Running it on a busy production system could bring that system to its knees. Your code needs help.

Start by debugging your malloc calls (calloc() calls malloc() )
There are good malloc debugging libraries out there for linux for example:
Dmalloc - Debug Malloc Home Page This will show where your problems occur.

Jim, thanks for reply.

I can not add swap on this station.
I succeeded to stop the program just before the exit out and run /bin/top - to see how memory it consumes: VIRT=79248, RES=28m
when there is a lot of free memory: ~400MB

It looks so:


%> top
Mem: 8017116k total, 7601900k used, 415216k free, 21300k buffers
Swap: 2097144k total, 184k used, 2096960k free, 2211488k cached

14315 baruchgu 25 0 79248 28m 472 R 99.8 0.4 0:43.47 vgen



Why malloc may not allocate more?
Again - it runs fine on five other linux stations.

note: I go to see into Dmalloc

Hi , experts.

I work on Linux station (RedHat 5.7), regular user, but have a root password.

Code:

%> uname -a
Linux ran1log06 2.6.18-238.1.1.el5 #1 SMP Tue Jan 4 13:32:19 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
%> cat /etc/issue
Red Hat Enterprise Linux Client release 5.7 (Tikanga)
Kernel \r on an \m

My C code application allocates memory by a lot of calloc(...)

The application fails when the station has many processes - calloc returns NULL pointer and error "Cannot allocate memory"
I see in /bin/top, that my program holds ~30MB RES memory and ~80MB vitual.

But there is a lot of free memory here.

Code:

%> /bin/top
.....
Mem:   8017116k total,  7977268k used,    39848k free,   223948k buffers
Swap:  2097144k total,      184k used,  2096960k free,  3070888k cached

there is no problem with limit also:

Code:

%> limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize unlimited
coredumpsize unlimited
memoryuse unlimited
vmemoryuse unlimited
descriptors 1024
memorylocked 32 kbytes
maxproc 65535

It runs fine in other not loaded Linux systems, also Red Hat 5.7

What may be the problem here?
May be some limit on the process on this system?
Or gcc compiler limits it?
How can I check it?


Note: I have already ported this problem in "Linux" forum. But got no help till now.

Think of process heap memory as a chessboard - one area made up of 64 small blocks.
suppose each one of those is 1 MB.

You make 64 calls to calloc, get 64 pointers to 1MB each. Next you free 10 of them at random.
How many will contiguous? Probably none. Will you have four adjacent squares. Very probably: No.

Now what happens when you want to calloc 4MB? You have 10MB free but it is not contiguous. Some versions of malloc will barf on this, some will call brk() to get an additional n MB from the OS. This is heap fragmentation. You now know that there are several different versions of malloc code. When you have malloc problems like you are having:
1. Try a malloc debugger
2. See what malloc you are using, change it if need be.


sysadmins can control during system installation and later system setup which version(s) of malloc are on a system, and which version of malloc is used by default during linking. So can other programmers.

Different versions of gcc also have different malloc implementations.

This is hypothesis you get to test with dmalloc. And other tools.

Code:

# show if there is library interposition

echo $LD_PRELOAD  # during runtime

# see what .so files you are really linked against
ldd compiled_program

# gcc version - note: this tells you how gcc was compiled which affects your problem.
gcc --version

Each of these results suggests a different workaround.
Your problem box may show something different than what you see elsewhere.

Did you compile locally or port the executable directly?

That is incorrect. The last post in your last thread isn't even yours. You already got answers, suggestions -- you just didn't like them.

In any case, we can hardly help you without seeing your code.

Why not post it in your original thread?