Login or Register to Ask a Question and Join Our Community


Solaris

ZFS Filesystem

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Solaris ZFS Filesystem
# 1  
Old 08-24-2015
ZFS Filesystem
Hi,
Recently we have new server T5 Oracle. We set up it for our database. For out database files we set one zfs filesystem. When i use iostat -xc the output as below. As you see the value for vdc4 is quite high.

Code:
                 extended device statistics                      cpu
device    r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b  us sy wt id
vdc0      0.6    3.9   10.8   37.5  0.0  0.0    1.9   0   0  10  7  0 83
vdc1     12.9    2.6 1644.2  309.9  0.0  0.1    7.6   0   1
vdc2      9.5    2.8 1208.8  351.9  0.0  0.1    8.4   0   1
vdc3      0.2    2.4   11.9   38.1  0.0  0.0    1.9   0   0
vdc4    266.6   83.1 32967.7 7561.5  0.0  3.2    9.1   0  65
vdc5      2.4    3.3  301.1  378.2  0.0  0.1   12.6   0   1
vdc6      5.8   52.1  715.3  718.0  0.0  0.1    2.4   0   6
vdc7      3.9   52.1  474.5  717.9  0.0  0.1    2.1   0   6
vdc8      0.0    0.0    0.0    0.0  0.0  0.0    2.3   0   0
nfs1      0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0

When i look on the output for memstat. The ZFS filesystem taking high mem percentange.

Code:
> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     355350              2776    8%
ZFS File Data             1660358             12971   40%
Anon                      1874388             14643   45%
Exec and libs               12338                96    0%
Page cache                 176508              1378    4%
Free (cachelist)             6483                50    0%
Free (freelist)            108879               850    3%
Total                    4194304             32768

Is this normal? When we database full backup the db will hang although the server load is normal during backup. Is that maybe related to the zfs filesystem setting? Hope can engligthen me on these.
Last edited by Don Cragun; 08-24-2015 at 01:37 AM.. Reason: Change ICODE tags to CODE tags, add ICODE tags.
# 2  
Old 08-24-2015
It has been a long time since I worked with a ZFS filesystem, but I don't think it is unusual for ZFS to consume memory that is otherwise unused as a cache for ZFS disk data.

Reading 33Mb/s and writing 7.5Mb/s may seem high, but with 0 wait time on the device, it doesn't appear to be a problem.

Are you seeing a high swap rate (or any indication that running processes are running poorly due to a lack of available memory)?
# 3  
Old 08-24-2015
It is not unusual for ZFS to eat almost all available memory.
You don't want that with database. Even if you are running on ZFS filesystems.

I would not recommend running databases on ZFS filesystems, since it requires alot of tuning to get it right. There is also an unresolved issue of fragmentation and for large implementation i would avoid ZFS for DB. ASM is the law Smilie

Are those FC or internal disks ?
What is the patchset you are running at (hypervisor & ldom - since i see it is a ldom) ?

Can you please tell what are the values kernel parameters :

Code:
ssd:ssd_max_throttle
zfs:zfs_vdev_max_pending
zfs:zfs_arc_max

Can you post output of following command during the problem ?
Code:
sar -d 2 10

Take a look at the avque, i suspect it is very high during the non responsive period.
If not, possibly your issue resides with arc_max (confirm that the machine is not swaping as Don suggested). Lower it to a sane value so your database doesn't run out of PGA space (it will start swapping then, causing extreme slowness).

Quote:
For out database files we set one zfs filesystem.
This is wrong, please take a look at the following documentation, and read it well :
Tuning ZFS for Database Products - Oracle Solaris 11.1 Tunable Parameters Reference Manual

In short, you will need multiple zpools on different spindles with different setups for various DB functionality (REDO, ARCH, DATA) and keep them under 80% (this is very important).
# 4  
Old 08-24-2015
Hi Peasant,

We are using NetApps SAN external storage. Actually we are using multiple zpools for our our database.

The values for our kernel parameters are also below;

Quote:
zfs_vdev_max_pending = 0xa
zfs_arc_max = 0x0

How to get the ssd:ssd_max_throttle value?

The output of the sar -d 2 10 as below;

Code:
17:18:59   device        %busy   avque   r+w/s  blks/s  avwait  avserv
17:19:01   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       5      16     0.0     0.8
           vdc2             79     7.8     363   87485     0.0    21.6
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     383   95701     0.0    26.1
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0
17:19:03   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              3     0.1      39     502     0.0     3.3
           vdc2              0     0.0       0       0     0.0     0.0
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     555  141827     0.0    18.0
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0
17:19:05   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       0       0     0.0     0.0
           vdc2             79     7.8     470  119166     0.0    16.7
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     448  114029     0.0    22.3
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0
17:19:07   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       0       0     0.0     0.4
           vdc2            100    10.0     528  133947     0.0    18.9
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     358   89443     0.0    27.9
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0
17:19:09   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       0       0     0.0     0.0
           vdc2            100     9.6     589  144250     0.0    16.2
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     434  109155     0.0    23.0
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0
17:19:11   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       0       0     0.0     0.0
           vdc2            100    10.0     658  167231     0.0    15.2
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     379   97137     0.0    26.3
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0
17:19:13   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       0       0     0.0     0.0
           vdc2            100    10.0     585  148283     0.0    17.1
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     424  108129     0.0    23.6
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0
17:19:15   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       0       0     0.0     0.0
           vdc2             14     1.2      95   14436     0.0    12.8
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     462  118123     0.0    21.6
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0
17:19:17   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       0       0     0.0     0.0
           vdc2             40     3.9     225   56873     0.0    17.3
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     485  122068     0.0    20.5
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0
17:19:19   nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       0       0     0.0     0.0
           vdc2            100    10.0     542  137370     0.0    18.4
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     459  109974     0.0    21.7
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0

Average    nfs1              0     0.0       0       0     0.0     0.0
           vdc0              0     0.0       0       0     0.0     0.0
           vdc1              0     0.0       4      52     0.0     3.0
           vdc2             71     7.0     405  100893     0.0    17.3
           vdc3              0     0.0       0       0     0.0     0.0
           vdc4            100    10.0     439  110551     0.0    22.8
           vdc5              0     0.0       0       0     0.0     0.0
           vdc6              0     0.0       0       0     0.0     0.0
           vdc7              0     0.0       0       0     0.0     0.0
           vdc8              0     0.0       0       0     0.0     0.0

We are using guest domain (VM). Any advise?
# 5  
Old 08-24-2015
Maybe a good starting point:

http://www.oracle.com/technetwork/se...-wp-167894.pdf
# 6  
Old 08-24-2015
Quote:
Originally Posted by Peasant
It is not unusual for ZFS to eat almost all available memory.
There is a lot of misunderstanding around this topic. All file systems will eat as much memory as they find useful, not just ZFS, unused memory being wasted memory anyway.

The big differences are:

- ZFS memory, including the ARC, is reported as used/unavailable while other file systems memory, the buffer cache and the page cache, is reported as free/available.

- ZFS memory is released asynchronously and gradually by observing RAM demand while other file system's memory is released synchronously and (almost) instantaneously. Where that matters is when an application requests a very large amount of non pageable memory as the allocation might fail. The arc_max tuning prevents ZFS to use all the RAM helping these allocations to succeed.
# 7  
Old 08-24-2015
also. snapshots. minimize them. more snapshots will mean more i/o. I had issues similar to this a while back and it all came down to snapshots and zfs_arc_max.
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to finish expanding a zfs filesystem?

I have a esxi 6.7 server running a Solaris 10 x86 vm (actually a bunch of them). The VM uses zfs for the pools (of course). I expand the underlying ESX logical disk, for example from 50GB to 100gb, then I set autoexpand=on <pool> that belongs to the esx logical disk. what am i missing to... (2 Replies)
Discussion started by: mrmurdock
2 Replies

2. Solaris

Extend zfs storage filesystem

Hello, Need to ask the question regarding extending the zfs storage file system. currently after using the command, df -kh u01-data-pool/data 600G 552 48G 93% /data /data are only 48 gb remaining and it has occupied 93% for total storage. zpool u01-data-pool has more then 200 gb... (14 Replies)
Discussion started by: shahzad53
14 Replies

3. Solaris

Please explain why ZFS is said to be a hybrid filesystem and a volume manager also

Hi guys! How come ZFS is said to be not just a filesystem but a hybrid filesystem and also a volume manager? Please explain. I will appreciate your replies. Hope you can help me figure this out. Thanks in advance! (1 Reply)
Discussion started by: Klyde
1 Replies

4. Filesystems, Disks and Memory

Howto Convert a filesystem from Veritas to ZFS?

Hi Folks, Looking for info here more than any actual HowTo, does anyone know if there is an actual way of converting a Veritas or UFS filesystem to ZFS leaving the resident data intact. All that I have been able to find, including the commercial products seem to require the FS backed up from... (1 Reply)
Discussion started by: gull04
1 Replies

5. Solaris

Mount old zfs filesystem

Hey all, I have a machine with 16 drive slots. Two of the drives have a ZFS mirror of the operating system, the other 14 contain the storage raidz. So, after installing Opensolaris on the OS drives, how can I remount the storage raid? TIA (11 Replies)
Discussion started by: PatrickBaer
11 Replies

6. Solaris

Why does the # of blocks change for a file on a ZFS filesystem?

I created a zpool and zfs filesystem in OpenSolaris. I made two NFS mount points: > zpool history History for 'raidpool': 2009-01-15.17:12:48 zpool create -f raidpool raidz1 c4t1d0 c4t2d0 c4t3d0 c4t4d0 c4t5d0 2009-01-15.17:15:54 zfs create -o mountpoint=/vol01 -o sharenfs=on -o... (0 Replies)
Discussion started by: sqa777
0 Replies

7. Solaris

How do I export a zfs filesystem that I created?

I created a zpool and two ZFS volumes in OpenSolaris. I would like both ZFS volumes to be exportable. However, I don't know how to set that up. These are the steps I did: 1) Create the zpool using raidz1 across five disks. I have six disks and created a zpool across 5 of them. c4t0d0... (3 Replies)
Discussion started by: sqa777
3 Replies
Login or Register to Ask a Question