Solaris

I was hoping to test a bug posted by oracle, which indicates that the system may crash when using a zfs volume as a swap device. We moved swap away from zfs and over to raw disk devices in order to stop the crashes we've been experiencing.
What I wanted to do was to re-create the crash in a test environment in order to validate our suspicions. I compiled a program that uses malloc() to eat up memory, but the system will kill it before any swap space is used.
how can I reliably force the system to use swap space?

Read the documentation about "sunvts".

SunVTS 7.0 Tests

Can you elaborate ? Solaris shouldn't kill processes reserving memory. There is no OOM killer mechanism.

here's some more details about what we run into. perhaps we're just not going about it the right way. for starters, we're running solaris 10 u10. installed ram is 128GB. Swap space is 8GB volume.
the idea is to trigger swap usage, and we try to monitor that two ways, using top and "swap -l"
first step that we try is to fill up /tmp with a couple of simple "mkfile 10g /tmp/memeater01" style commands. We can see the available ram decrease each time we create one of those files in /tmp.

we get down to about 10-ish GB of free ram and start to encounter "no space available" errors when trying to create tmp files.

Next think we tried was to push it over the edge using a program that simply uses malloc() calls to reserve ram and hold onto it. we can also watch the avalable ram decrease until the 10GB (roughly) threshold and then this program dies.

A combination of both /tmp files and the malloc() program still dies around the 10gb free threshold and we never ever see any indicators that swap space is being used.

---------- Post updated at 10:29 AM ---------- Previous update was at 10:24 AM ----------

Just downloaded VTS, and started reading the documentation. Tried a test run, but don't know if the defaults are what we need. We only enabled the memory test and ran it. According to top, free RAM dropped to 36GB and stayed there, Free swap stayed the same no changes. We sat there and watched top for about 30 minutes before getting pulled away to a meeting. so we stopped it and decided to try later. are there some parameters we need to set or change?

Just calling malloc() isn't going to do a thing other than reserve swap space. The call to malloc() is just going to reserve memory - it's not going to actually create the virtual pages and their mappings to physical pages. For that to happen, you actually need to USE the memory.*

Change your malloc() calls to calloc() - you should see swap space get used then.

* - for the overly-pedantic, that's not TOTALLY true as multiple calls to malloc() will probably cause at least a handful of pages to be created and mapped, but it's close enough.

You have exhausted your virtual memory, which includes most of the RAM plus all of the swap area when you get these messages. Can you post the actual numbers and the commands you use to measure memory and swap usage ?
How (why) does it die ?
Are you you actually accessing all of the the allocated pages ? VTS is to stress test you hardware in order to detect faulty components. I don't think this is what you are looking for, although I don't precisely understand what exactly you are trying to achieve/fix ...

from the VTS docu:

from that information i suppose there will be access to the swap space...