Solaris

Shouldn't impact performance at all. Database IO isn't going to go through the ARC - it'll be either direct IO or synchronous. Log files are generally streamed and forgotten about, so not having cached data from writes there isn't a big deal. And once the cache gets beyond a few tens of MB, the effective filesystem cache hit rate isn't going to change much anyway given the usage patterns on a pure DB server.

In my experience, performance often gets better because there's actually some free RAM on the server, so response to transient demands is a helluva lot faster.

If you want the DB to cache data, create a larger-than-default SGA for that, and create larger buffer and redo log pools in it as needed.

To get the max performance out of an Oracle DB running on Solaris, you really do have to get the ZFS ARC out of the way. (And you also have to be really careful about how your DB job processes behave - you do not want to have your DB trying to start or stop several thousand processes all at the same time...)

(I spent a few years consulting for a customer using multiple large Oracle RAC clusters on SPARC servers - one of my main jobs was getting the best possible performance out of the servers. Oracle on Solaris is as good as it gets for performance and reliability - yes, better than Linux for a lot of reasons - but there are some quirks - and the ARC is one of them.)

The metadata will still need to be cached though. Note that zfs_arc_max is being deprecated with Solaris 11.2. The new dynamically tunable user_reserve_hint_pct allows much more flexibility.

Hi achenle/jlliagre,
Thanks for feedbacks. I really appreciate it. The vendor also proposing on limiting the zfs_arc parameter value. Currently I am trying use perfstat script to capture log when DB backup to find out the culprit. Will share here once I got the output.
Is that zfs_arc_max =0*0 mean unlimited?

0 means there is no user configured limit.

In such case, the OS is limiting the ARC size to either 75% or the RAM if the machine has less than 4 GB of RAM, and "total RAM minus 1 GB" otherwise.

Hi Jlliagre,achenle,
I have set the zfs_arc_max = 4GB but still when run the backup the DB seems hangs and got error "WARNING: aiowait timed out 1 times". The memstat during backup as below;

Code:

 Page Summary                Pages                MB    %Tot
------------           ----------------  ----------------  ----
Kernel                     291354              2276           7%
ZFS File Data              519224              4056       12%
Anon                      1900341             14846        45%
Exec and libs               13193               103         0%
Page cache                    612                 4            0%
Free (cachelist)             8058                62          0%
Free (freelist)           1461522             11418      35%
Total                       4194304             32768

Aggressively reducing the arc size likely affected ZFS performance. 11 GB were unused, i.e. wasted, when the statistic was captured. You should monitor the memory usage during a long enough period of time to figure out the system peak memory needs.

4 GB cache is more than enough. I seriously doubt it impacted performance.

How are the backups being run? And what exactly is emitting the "aiowait timed out" errors?