If you don't have access to the server, it is not obvious how you might with the problem. You (or someone) will need a service contract to obtain patches, so I would advise using that contract to open a call with HP and ask them to suggest patches for your version of HP-UX that might help with a mmap problem.

Anyway, the man page says " [ENOMEM] ... Or, MAP_FIXED was not specified and there is insufficient room in the address space to effect the mapping." So if it's not an OS problem, it is probably a memory leak. Maybe you could add some code to induce a core dump when mmap returns ENOMEM, then examine the core dump to see where the virtual addresses went. (But if you can't access the server, I do not know you might examine a core dump.)