Most of the ZFS ARC on Oracle DB servers is log file data that will never be read back. It'll just sit in the ARC until something else needs the RAM - there's no point in caching that data. I bet the cache hit rate is still > 99%.
Brendan's blog » Activity of the ZFS ARC
Anything over 95% doesn't really improve performance.
I'm thinking however the OP is doing backups is causing ZFS to block IO for a while, thus the DB aio_wait() calls time out. Maybe the backup process used is to take a snapshot of the ZFS file system will the DB is active, and then use ZFS send on the snapshot? (Although that really won't produce a viable backup of an Oracle DB....)
As I posted earlier, if you want to use the memory to speed up an Oracle DB, use it as the DB's SGA.