I have a few hundred oracle boxes - in my experience the systems are most comfortable when comp (the avm value in vmstat x 4k) doesnt exceed 80% as this leaves enough memory for all the oracle forked processes, IO buffering, batch processing and so on.
When my memory utilization exceeds these 80% than my system starts scanning / freeing memory which utilizes cpu and slows down the DB as the system waits for sufficient freed up memory to continue processing - which obviously is bad. The higher the scan to free ratio - so the more pages need to be scanned to free up the memory I actually need for the given workload - the slower the system gets and the more cpu is utilized. So I make sure I always have plenty of memory - as particularly for oracle the need of non-comp memory is very valid as its usually a filesystem based DB - and not finding filecache if needed slows down the DB too as no IO can happen ...
Please note - during rman backups you still will see some scan and free as this puts - at least in my environments - a large amount of additional load onto the systems. So my 80% are during busy times but not when rman runs. Nmon is pretty helpful to find out what is good for your system and when you do have your busy times.

Virtual memory btw is physical memory + pagingspace in 4k pages. Virtual memory in use is how much of this you are actively using - ideally visibly less than you physically have

Regards
zxmaus