Solaris

You were still focused on RAM with memstat while the issue was about Virtual memory. Again, while I agree adding RAM or freeing RAM previously allocated for ZFS in increasing the size of free virtual memory, failing to observe the virtual memory usage didn't help. Something was reserving most or all of virtual memory available on your system, identifying what was doing it might have help understanding why.

Give it a test on 11.3, with 20ish GB of ram.

Download and create repository, pulling that 140 GB all version oracle repository will do
Run a simple find against repository directory with redirection to a file.
This is a great test case since that repo contains a lot of files.
Observe the memory footprint increase, mostly going to kernel and zfs lines in memstat.
A new ::komstat will output stuff more closely.

After it is finished run a small c program allocating memory.
System will pause literally for a time until reap process is complete.
This of course, can cause cluster failures, database failovers, nfs client issues , kernel panics and similar.
Dangerous stuff, even from a security standpoint.

Those are bugs present in oracle site, keyword kom reap
If i recall correctly those are fixed in higher versions, but i have yet to test it.

In my opinion, keeping limit and/or sizing of this zfs / kernel behavior is a must under 11.3 under any conditions on any patchset.

Regards
Peasant.