NJAMD - Not Just Another Malloc Debugger
void *calloc(size_t nmemb, size_t size);
void *malloc(size_t size);
void free(void *ptr);
void *realloc(void *ptr, size_t size);
new, new, delete, delete
char *strdup(const char *s);
And so much more...
kill -USR1 <pid>
This manpage describes the library subsystem of NJAMD, which can be used standalone, with
the front end (console only, and needs work), from within gdb(1) or from within any other
debugger. It also comes with a very useful utility njamdpm(1) that allows you to do post-
mortem heap analysis.
NJAMD is a full featured malloc debugger. That is, it protects against all common dynamic
memory bugs, including overflow, underflow, writes to freed memory, and memory leaks, all
without recompiling or even relinking your executable. In addition, it is able to trace
memory leaks even through arbitrary library functions that wrap malloc(3), such as
strdup(3), GUI widget allocators, and even C++ new and delete.
Normally, when a program does something illegal with its dynamic memory (such as writing
past the end of a buffer returned by malloc(3), ie: an overflow), its execution may not
immediately terminate. Instead, bizzarre and unexpected results can occur later on during
program execution. This is due to the fact that malloc implementations store book keeping
information before and after allocated segments. So overwriting these regions won't cause
your program to crash right away, but will cause chaos during subsequent malloc requests,
or even during usuage of memory returned from otherwise seemingly valid malloc()
NJAMD changes all this. It provides immediate notification (through segmentation fault) if
you do anything illegal with your memory. Using your favorite debugger, you can pinpoint
the source of error to the line, and even to the assembly instruction.
With one exception, the behavior of the debugger is controlled entirely through the fol-
lowing environment variables. In fact, the debugger needn't even be linked to your program
on systems that suport the LD_PRELOAD environment variable. Unless otherwise stated, these
environment variables are tested .B for existance only. That is, doing something like
export NJAMD_ALLOW_READ=0 will still allow read access past the ends of buffers.
This environment variable instructs the dynamic linker to override system supplied
functions with those in a specified library, namely ours. To use LIBNJAMD to debug
your programs, enter the equivalent to
into your shell. All subsequent programs run from that shell will then use LIBN-
JAMD's allocator routines instead of those in the standard libc.
Alternatively, to debug only one command, enter
NOTE on Irix systems, the variable is _RLD_LIST and the syntax is _RLD_LIST=libn-
jamd.so:DEFAULT. Otherwise the behavior is the exact same.
To use LIBNJAMD to protect against overflows (accessing memory past the end of an
allocated buffer), enter the equivalent to
into your shell. This is the default mode of operation.
There are two ways to protect against underflows (accessing memory before an allo-
cated buffer), strict and weak. Weak is considerably faster than strict, and uses
half as much memory. However, weak will only catch underflows greater than 4 or 8
bytes, depending on your archetecture.
To protect against ALL underflows, enter
To protect against most larger underflows, enter
For memory leak checking only, enter
This option uses standard libc malloc, and is thus is much faster and lighter than
the other options, for people who just want memory leak accounting. Note that it is
unavailable on platforms that don't support dlopen(2). Also, this option will mis-
report leaked memory by one malloc on some platforms (GNU/Linux w/ glibc 2.1),
becuase malloc calls itself to set up some data structures.
Do note that each version of the library performs consistancy checks so that you
know if the opposite error occured when you try to free that block. For example,
when you free a buffer, the overflow version checks to make sure that the data
before your buffer hasn't changed, and the underflow versions checks to make sure
that the data after your buffer hasn't changed. So at the worst, you always know of
a memory error by the time you free the memory. This even applies to the "none"
There are various methods of handling accesses to freed memory, and each has its
advantages and disadvantages. If you select NJAMD_PROT=none this setting has no
effect (NJAMD operates as if you selected no free protection).
The default method is to protect freed memory. A double free will yield a segmenta-
tion fault and no error message, and any access to a memory region freed by free or
realloc will cause a segmentation fault.
This option both protects freed memory and provides you with some sort of notifica-
tion when you try to free a chunk twice. While this causes no physical memory loss,
it does pollute the address space a bit, and can bog down the operating system ker-
nel with excessive mappings to keep track of. The BSD's especially are hurt by this
option, and Linux has a limit of 65536 mappings, which can be used up pretty
This method provides no protection of freed memory. Writes to freed memory may pro-
duce the same bizzare and unpredicatble results as when using a normal malloc
implementation. In addition, this option is 10 times faster than the other error
checking schemes due to a free-list cache put in NJAMD > 0.9.0 (In fact,
NJAMD_PROT=over with no free checking is FASTER than NJAMD_PROT=none for the
Use of this option is recommended when allocation intensive progams run out of mem-
ory under either of the preceding options, or if NJAMD is prohibively slow. Do
note that by default, some OS's do not allow you to map the entire address space.
You must use sysctl(2) to allow this. In Linux, for example, you must issue
sysctl -w vm.overcommit_memory=1
to use the entire address space. Try doing this instead of turning off the checking
of freed memory. Also, look into the kernel_mods patches that came with the source
In addition, if you are trying to use this option because you are running out of
ADDRESS space, you might want to try the next option: (NJAMD_NO_FREE_INFO)
Setting this option will cause NJAMD to neglect to free ANY memory. I can't think
of any circumstances where this would be useful, but it was very easy to implement
Setting this option will cause NJAMD to internally conserve heap table space by not
recording information about freed segments. Translation, things will go quicker and
take less memory in some cases, but you will lose information about where segments
are freed. This option may be necessary to debug HUGE applications on 64bit procs,
when using NJAMD_PROT=none
In order for programs to work correctly under certain archetectures (ie, sparc and
most other RISC CPUs), malloc must return memory aligned to a certain number of
bytes if you want that section of memory to contain pointers and floating point
values. The alignment of your archetecture is detected automatically when you
install NJAMD. However, note that aligning memory to n bytes will cause the over-
flow detection to miss overflows of up to n bytes. If you are on a RISC CPU, but
know that alignment is not an issue in your program (ie, if it only deals with
export NJAMD_ALIGN=1 to set alignment to 1 byte.
Setting this option instructs LIBNJAMD to dump memory leak diagnostics to the front
end (or standard error when running standalone) upon program termination. num lev-
els of stack trace are provided for each malloc and free (the default max is 3, and
can be set at NJAMD compile time in ./include/lib/njamd.h, via the TRACE_DEPTH
Do note that it is common practice for short-lived programs such as ls(1) to simply
exit without freeing memory.
Setting this environment variable instructs NJAMD to dump a short summary of memory
usage versus address space usage. This option will help you figure out how much
overhead is being used by NJAMD, and how much address space in total was needed to
debug your application. In other words, it will either give you an excuse to buy a
64 bit CPU, or a few more RAM chips ;)
By default, NJAMD will catch all deadly signals in order to perform cleanup, pro-
vide statistics, and give its own backtrace of when the fault occurred. However,
when you are using a debugger, this behavior is not always desirable.
If you would like NJAMD to perform cleanup and statistics information, but would
also like a core file, then
Using soft core dumping will cause the return address information INSIDE the core
file to make no sense, but it will allow NJAMD to provide a call stack dump upon
exit. Using softcore also limits the coresize to 4megs (defined through
NJ_LIMIT_SOFTCORE in the source).
If you would like the core file to be perfectly valid and complete at the expense
of statistics and post-mortem heap integrity, then
Setting this option will cause the program heap to remain after execution in a file
forrmatted ./njamd-<pid>-heap. Unlike other malloc debuggers, saving the heap in
NJAMD imposes no extra performance overhead on the system. The heap is always
mapped to a file in tmp, but this file only persists if this option is set. To uti-
lize this file, use the njamd post-mortem utility njamdpm(1). Be advised that the
heap can take up as much as 8 megs on 32 bit systems. ls may report it as 8 megs
due to the lseek. The actual size is much less. Use du(1)
NJAMD's default action is to ignore shared library return addresses and only give
you return addresses in the statically linked portion of the program. This can be
a problem if your program consists of a large amount of supporting libraries, and
the static section is simply a main loop. So instead, to provide return addresses
in the libraries, set
When debugging programs that use libraries compiled with optimization greater than
-O2 or with -fomit-frame-pointer, you must disable tracing, or NJAMD will segfault.
This isn't really NJAMD's fault. Gcc (see 'info gcc') claims that if
__builtin_return_address is unavailable it will just return NULL. Instead it seg-
mentation faults. I've notified the gcc team, but received no response.
Glibc versions up to and including 2.1.2 had a bug relating to the sscanf(3) code
that was tripped by this library. As much as we would like to, unfortunately, we
cannot take credit for being the first to discover this bug. The glibc folk found
and fixed the bug in version 2.1.3. At any rate, this option is provided as a work-
around. It allows reads past the end of a buffer to not segfault your program. The
default is to forbid read, write, and execute attempts.
ANSI C specifies that malloc's of 0 are legal, and that free's of NULL are illegal.
However, NJAMD's default action is to warn when either takes place, as either usu-
ally indicates a bug (or at the very least, an unneeded call). To turn this behav-
ior off for either free or malloc, issue
export NJAMD_ALLOW_FREE_0=1, or
export NJAMD_ALLOW_MALLOC_0=1 respectively.
Dumping memory information
Issuing a kill -USR1 to a program running under LIBNJAMD will cause memory usage diagnos-
tics to be dumped to standard error. These are human readable lines of the form
njamd: Memory leak of XX bytes allocated at address 0xXXXXXXX.
The address given is the actual address of the corresponding memory allocation in your
code. In future versions of NJAMD, scripts or runtime functionality will be provided to
translate these addresses on the fly to functions and line numbers. Until then, read on to
find out how to use gdb(1) to translate them for you.
Debugging from within gdb
Using LIBNJAMD from within gdb is simple. The command to set environment variables in gdb
is set env VARIABLE=value. So, to instruct gdb to use LIBNJAMD, issue
set env LD_PRELOAD=libnjamd.so
from INSIDE gdb. Issuing an LD_PRELOAD command to your shell before starting gdb causes
gdb to use that library as well, which means gdb would be using NJAMD, and unless you're
on the gdb development team, you're probably not interested in debugging gdb :)
All other options can be set in the same mannor from with gdb, or in the shell's environ-
ment outside gdb.
You can obtain memory leak information at any point by setting a break point, and then
issuing signal SIGUSR1 at that breakpoint. This will provide a memory leak dump as
described above. In addition, so will
You can also get information about any address used by njamd by issuing call
This will dump out a call stack of the malloc or free that contains that address. This is
very useful for gaining information about a segmentation fault. Ie if the segfault occurs
on a line that does buf[i] = 2, issue
call __nj_ptr_info(&buf[i]) to gdb.
To get gdb to translate these return addresses into something meaningful, issue
info line *0xaddress to obtain the line number of the allocation request, or
list *0xaddress to see the adjacent code as well.
Another neat trick you may find handy for tracking down things like free(NULL) and mal-
loc(0) and other behavior that produces warnings is to set a breakpoint at __nj_eprintf so
that you can determine the location of the offending instruction. __nj_eprintf is NJAMD's
general purpose error function. It is called to print out any NJAMD warning or error mes-
sage you see. Note that if you are using LD_PRELOAD, you may have to set a breakpoint in
main and start the program before setting this breakpoint for gdb to know that
__nj_eprintf is a valid symbol.
If anything goes wrong, please read the NOTES file in the NJAMD source directory, ESPE-
CIALLY before reporting any bugs. Many platforms and compiler flag combinations cause
problems. Please read that file for more info. Also, whenever sending me a bug report,
please send a self-contained code snippet to reproduce the bug.
If you notice large performance problems, especially on the xBSD's,
Setting the checking of freed memory to none allows for an optimization that can speed up
NJAMD 10 fold, making it comparable to libc malloc in some usage cases!
Furturemore, by default NJAMD uses the minimum alignment possible that won't cause a seg-
fault. Increasing this alignment may increase performance on some archetectures, because
you won't lose cycles for unaligned memory accesses. Try setting
export NJAMD_ALIGN=4 or 8
Also, there is a define at the top of ./include/lib/njamd.h in the NJAMD source tree,
TRACE_DEPTH, that allows you to set how large of a stack trace is recorded. Lowering this
to 1 (default is 3) may speed things up a bit, as well as cut the size of the heap file
almost in half.
I've discovered that the limit of 65536 mappings in Linux includes permission variations.
In other words, you will not be able to debug large apps due to this limit. To get around
this, apply either fix_map-<kernel_version>.patch or proc_map-<kernel_version>.patch to
your kernel, and recompile. See the ./kernel_mod/README file for more info.
Core dump sizes are limited to 4 megs (and are probably next to useless) because of the
huge amount of mapping that takes place. Some OS's (again, those evil BSD's ;) actually
zero-fill mapped but unfaulted memory as it is dumped to disk, causing a core dump to take
a horrendous amout of time and disk space.
For information on how the system works, read the programmers documentation in the source
tree, and check out my (admittantly incomplete) Shared Memory HOWTO:
Mike Perry <email@example.com> - libnjamd
Steve Engelhardt <firstname.lastname@example.org> - Front End
njamdpm(1), efence(3), malloc(3), mmap(2), mprotect(2)
NJAMD - 5 Oct 00 LIBNJAMD(3)