Available virtual memory = RAM + swapfile - (currently in use)
So, if the total is 3GB + (probably) 6GB swap, you have nine GB to use. The total memory for a process is kernel space + user space. What version of unix do you have?
For example, free gives you this information on Linux boxes.

Generally, processes that use huge amounts of memory will cause the system to bog because of all the extra paging activity involved. If this is a multiuser system, you might want to ask first. And check ulimit for your account, also.