If this would be my cluster, I would try to switch on async IO (not sure why it is off on your box as the default is on).
I would like to see your output of vmstat -s
I completely agree with Bakunin that your memory is overcommitted. Your AVM value is way too high, your freelist way too low for a DB box. IBM recommends an avm value between 70 and 80% for a reason. DBs are doing the 'work' in memory - not on disk. The smaller the free list, the more the system has to do - as the read and write from memory to disk are much higher than necessary - in an ideal world the memory holds the database - or at least the parts that are frequently accessed. That this is not the case on your system are shown by the very high system cpu usage levels.
... The sy column details the percentage of time the CPU was executing a process in system mode. ... reading or writing of a file requires kernel resources to open the file, seek a specific location, and read or write data, unless memory mapped files are used...
For me it looks like your system is artificially write intense - and that it would be much calmer if it would have the memory it would like to have for smooth operations. If it would be my box, I would add memory until your free list goes into the 6 digit area at least - you will most likely experience a significant drop in system cpu usage.
Is the output you have shown us from vmstat taken at a rather busy or rather idle time?
I cannot see enough processes in your runqueue that would even justify 20 cores - why do you have 40? And how many of them are folded all day / how many really ever used?
Kind regards
zxmaus