I can't help you with the specific kernel parameter since I don't know HPUX very well. You'll have to ask your DBA or look it up.

But, I think I know what is going on. Oracle sets its shared memory area when it starts up. Even if it looks to the OS like there is 5GB free, if you take any of it which Oracle expected to be able to use Oracle will complain. If you are setting aside some memory for caching as you described, you probably have to decrease the shared global area in Oracle to compensate.

Hope that helped, I know it wasn't long on details . . .