AIX

Hello folks,

I am facing a problem with NFS after doing the HACMP configuration

Issue : The showmount -e pr2serv at pr1serv ,it is only showing 1 mount exported /oracle , instead of showing all 7 as per /etc/exports file.

issue : while mounting /oracle at pr1serv getting permission denied error.


NFS Mounting

At pr1serv
# showmount -e pr2serv
export list for pr2serv:
/oracle pr1serv,px1serv
# mount pr2serv:/oracle /oracle
mount: 1831-011 access denied for pr2serv:/oracle
mount: 1831-008 giving up on:
pr2serv:/oracle
The file access permissions do not allow the specified action.

# cd /
# ls -ldr /oracle
drwxrwxrwx 7 root system 256 Jan 28 09:48 /oracle
# ps -ef | grep -i nfs
root 467122 348190 0 09:37:55 pts/0 0:00 grep -i nfs
root 565304 200840 0 Feb 05 - 0:00 /usr/sbin/nfsd 3891


At pr2serv
More /etc/exports

/oracle -sec=sys:krb5p:krb5i:krb5:dh,rw,access=pr1serv:px1serv,root=pr1serv:px1serv
/oracle/PR0 -sec=sys:krb5p:krb5i:krb5:dh,rw,access=pr1serv:px1serv,root=pr1serv:px1serv
/oracle/PB0 -sec=sys:krb5p:krb5i:krb5:dh,rw,access=pr1serv:px1serv,root=pr1serv:px1serv
/oracle/PX0 -sec=sys:krb5p:krb5i:krb5:dh,rw,access=pr1serv:px1serv,root=pr1serv:px1serv
/oracle/PX0/102_64 -sec=sys:krb5p:krb5i:krb5:dh,rw,access=pr1serv:px1serv,root=pr1serv:px1serv
/oracle/PB0/102_64 -sec=sys:krb5p:krb5i:krb5:dh,rw,access=pr1serv:px1serv,root=pr1serv:px1serv
/oracle/PR0/102_64 -sec=sys:krb5p:krb5i:krb5:dh,rw,access=pr1serv:px1serv,root=pr1serv:px1serv

In case you export the NFS exports from within an HACMP cluster you should use an Alternate Exports File /usr/es/sbin/cluster/etc/exports.
Create exports with
# smitty mknfsexp
and name Alternate Exports File in this panel. (In your case probably just move the /etc/exports file to the new location). Make sure the Alternate Exports File exists on every cluster node (add to HACMP file collection).

PS Reading the NFS sections in the HACMP Installation and Administration Guide does not hurt
High Availability Cluster Multi-Processing (HACMP) publications - IBM e(logo)Server Cluster Information Center

it did solve the problem partially when I tried the suggested solution , now I can see all exported file system after restarting NFS.
But still NFS server is not responding while mounting ( exported file sys) on other node).

The strange thing I found that exported directory can be mounted on a node which is outside cluster but , between cluster node no mounting possible.
Please suggest.

On SERVER 1 : pr1serv

# hostname
pr1serv
# showmount -e pr2serv
export list for pr2serv:
/oracle pr1serv,px1serv
/oracle/PR0 pr1serv,px1serv
/oracle/PB0 pr1serv,px1serv
/oracle/PX0 pr1serv,px1serv
/oracle/PX0/102_64 pr1serv,px1serv
/oracle/PB0/102_64 pr1serv,px1serv
/oracle/PR0/102_64 pr1serv,px1serv
/sapbackup dr2serv
# showmount -e pr1serv
export list for pr1serv:
/usr/sap/PR0 pr2serv
/usr/sap/trans_PR0 pr2serv
/sapmnt/PR0 pr2serv,px1serv
# mount pr2serv^C
# ls -ltr oracle
total 24
drwxr-xr-x 2 root system 256 Jan 06 18:43 stage
drwxr-xr-x 14 root system 4096 Jan 06 18:43 PR0
drwxrwxrwx 3 root system 256 Jan 28 09:38 client
drwxr-xr-x 14 root system 4096 Feb 02 12:10 PB0
drwxr-xr-x 14 root system 4096 Feb 02 12:14 PX0
# mount pr2serv:/oracle /oracle
NFS server pr2serv not responding still trying



ON SERVER 2 : pr2serv
# showmount -e pr2serv
export list for pr2serv:
/oracle pr1serv,px1serv
/oracle/PR0 pr1serv,px1serv
/oracle/PB0 pr1serv,px1serv
/oracle/PX0 pr1serv,px1serv
/oracle/PX0/102_64 pr1serv,px1serv
/oracle/PB0/102_64 pr1serv,px1serv
/oracle/PR0/102_64 pr1serv,px1serv
/sapbackup dr2serv
# showmount -e pr1serv
export list for pr1serv:
/usr/sap/PR0 pr2serv
/usr/sap/trans_PR0 pr2serv
/sapmnt/PR0 pr2serv,px1serv
# hostname
pr2serv
# showmount -e px1serv
export list for px1serv:
/sapmnt/PB0 pr2serv,pr1serv
/sapmnt/PX0 pr2serv,pr1serv
/usr/sap/trans_PB0 pr1serv,pr2serv
/usr/sap/trans_PX0 pr1serv,pr2serv
/usr/sap/data pr1serv,pr2serv




# showmount -e pr2serv
export list for pr2serv:
/oracle pr1serv,px1serv
/oracle/PR0 pr1serv,px1serv
/oracle/PB0 pr1serv,px1serv
/oracle/PX0 pr1serv,px1serv
/oracle/PX0/102_64 pr1serv,px1serv
/oracle/PB0/102_64 pr1serv,px1serv
/oracle/PR0/102_64 pr1serv,px1serv
/sapbackup dr2serv
# showmount -e pr1serv
export list for pr1serv:
/usr/sap/PR0 pr2serv
/usr/sap/trans_PR0 pr2serv
/sapmnt/PR0 pr2serv,px1serv
# showmount -e px1serv
export list for px1serv:
/sapmnt/PB0 pr2serv,pr1serv
/sapmnt/PX0 pr2serv,pr1serv
/usr/sap/trans_PB0 pr1serv,pr2serv
/usr/sap/trans_PX0 pr1serv,pr2serv
/usr/sap/data pr1serv,pr2serv
# hostname
px1serv
#

I wonder if you could post the output of "cltopinfo " and "clshowres | grep -i nfs" from one cluster node plus the output of "netstat -rn" from both cluster nodes?

Hi there,



Please find the requested listing and suggest.

*****************
# ./usr/es/sbin/cluster/utilities/cltopinfo
Cluster Name: cluster1
Cluster Connection Authentication Mode: Standard
Cluster Message Authentication Mode: None
Cluster Message Encryption: None
Use Persistent Labels for Communication: No
There are 3 node(s) and 4 network(s) defined

NODE pr1serv:
Network net_diskhb_01
pr1_pr2_disk /dev/hdisk11
Network net_diskhb_02
pr1_px1_disk /dev/hdisk12
Network net_diskhb_03
Network net_ether_01
px1serv 192.168.117.211
pr1serv 192.168.117.201
pr2serv 192.168.117.206
pr1serv_boot2 192.168.123.201
pr1serv_boot 192.168.120.201
pr1serv_boot0 192.168.118.201

NODE pr2serv:
Network net_diskhb_01
pr2_pr1_disk /dev/hdisk11
Network net_diskhb_02
Network net_diskhb_03
pr2_px1_disk /dev/hdisk13
Network net_ether_01
px1serv 192.168.117.211
pr1serv 192.168.117.201
pr2serv 192.168.117.206
pr2serv_boot0 192.168.118.206
pr2serv_boot 192.168.120.206
pr2serv_boot2 192.168.123.206

NODE px1serv:
Network net_diskhb_01
Network net_diskhb_02
px1_pr1_disk /dev/hdisk12
Network net_diskhb_03
px1_pr2_disk /dev/hdisk13
Network net_ether_01
px1serv 192.168.117.211
pr1serv 192.168.117.201
pr2serv 192.168.117.206
px1serv_boot2 192.168.123.211
px1serv_boot 192.168.120.211
px1serv_boot0 192.168.118.211

Resource Group pr1serv
Startup Policy Online On Home Node Only
Fallover Policy Fallover To Next Priority Node In The List
Fallback Policy Never Fallback
Participating Nodes pr1serv pr2serv px1serv
Service IP Label pr1serv

Resource Group pr2serv
Startup Policy Online On Home Node Only
Fallover Policy Fallover To Next Priority Node In The List
Fallback Policy Never Fallback
Participating Nodes pr2serv px1serv pr1serv
Service IP Label pr2serv

Resource Group px1serv
Startup Policy Online On Home Node Only
Fallover Policy Fallover To Next Priority Node In The List
Fallback Policy Never Fallback
Participating Nodes px1serv pr2serv pr1serv
Service IP Label px1serv

Total Heartbeats Missed: 939
Cluster Topology Start Time: 02/10/2009 16:01:12

#
*****************
# ./usr/es/sbin/cluster/utilities/clshowres | grep -i nfs
Filesystems/Directories to be exported (NFSv2/NFSv3)
Filesystems/Directories to be exported (NFSv4)
Filesystems to be NFS mounted
Network For NFS Mount
Filesystem/Directory for NFSv4 Stable Storage
Filesystems/Directories to be exported (NFSv2/NFSv3)
Filesystems/Directories to be exported (NFSv4)
Filesystems to be NFS mounted
Network For NFS Mount
Filesystem/Directory for NFSv4 Stable Storage
Filesystems/Directories to be exported (NFSv2/NFSv3)
Filesystems/Directories to be exported (NFSv4)
Filesystems to be NFS mounted
Network For NFS Mount
Filesystem/Directory for NFSv4 Stable Storage

*****************

# netstat -rn
Routing tables
Destination Gateway Flags Refs Use If Exp Groups

Route Tree for Protocol Family 2 (Internet):
default 192.168.117.254 UG 0 104 et0 - -
10.222.113.192/27 192.168.117.252 UGS 0 21525 en0 - -
127/8 127.0.0.1 U 12 6322025 lo0 - -
192.168.1/24 192.168.117.254 UGS 0 18300 en0 - -
192.168.10/24 192.168.117.254 UGS 2 849839 en0 - - =>
192.168.10/24 192.168.117.254 UGS 0 0 et0 - -
192.168.117.0 192.168.117.202 UHSb 0 0 et0 - - =>
192.168.117.0 192.168.117.201 UHSb 0 0 en2 - - =>
192.168.117/24 192.168.117.202 U 3 440 et0 - - =>
192.168.117/24 192.168.117.201 U 22 148124316 en2 - -
192.168.117.201 127.0.0.1 UGHS 1 4805 lo0 - -
192.168.117.202 127.0.0.1 UGHS 0 3 lo0 - -
192.168.117.255 192.168.117.202 UHSb 0 0 et0 - - =>
192.168.117.255 192.168.117.201 UHSb 0 4 en2 - -
192.168.118.0 192.168.118.201 UHSb 0 0 en0 - - =>
192.168.118/24 192.168.118.201 U 2 4979522 en0 - -
192.168.118.201 127.0.0.1 UGHS 0 660404 lo0 - -
192.168.118.255 192.168.118.201 UHSb 0 11 en0 - -
192.168.120.0 192.168.120.201 UHSb 0 0 en2 - - =>
192.168.120/24 192.168.120.201 U 1 4032681 en2 - -
192.168.120.201 127.0.0.1 UGHS 0 655859 lo0 - -
192.168.120.255 192.168.120.201 UHSb 0 11 en2 - -
192.168.121.0 192.168.121.201 UHSb 0 0 en1 - - =>
192.168.121/24 192.168.121.201 U 0 0 en1 - -
192.168.121.201 127.0.0.1 UGHS 0 133 lo0 - -
192.168.121.255 192.168.121.201 UHSb 0 8 en1 - -
192.168.123.0 192.168.123.201 UHSb 0 0 en3 - - =>
192.168.123/24 192.168.123.201 U 2 4437278 en3 - -
192.168.123.201 127.0.0.1 UGHS 2 1181786 lo0 - -
192.168.123.255 192.168.123.201 UHSb 0 11 en3 - -

Route Tree for Protocol Family 24 (Internet v6):
::1 ::1 UH 0 128 lo0 - -
# hostname
pr1serv
#

**********************

# netstat -rn
Routing tables
Destination Gateway Flags Refs Use If Exp Groups

Route Tree for Protocol Family 2 (Internet):
default 192.168.117.254 UG 0 50 et0 - -
127/8 127.0.0.1 U 9 3334583 lo0 - -
192.168.10/24 192.168.117.254 UGS 2 1815830 en0 - - =>
192.168.10/24 192.168.117.254 UGS 0 0 et0 - -
192.168.117.0 192.168.117.207 UHSb 0 0 et0 - - =>
192.168.117.0 192.168.117.208 UHSb 0 0 et1 - - =>
192.168.117.0 192.168.117.206 UHSb 0 0 en2 - - =>
192.168.117/24 192.168.117.207 U 3 196 et0 - - =>
192.168.117/24 192.168.117.208 U 0 0 et1 - - =>
192.168.117/24 192.168.117.206 U 65 283327719 en2 - -
192.168.117.206 127.0.0.1 UGHS 1 1648029 lo0 - -
192.168.117.207 127.0.0.1 UGHS 0 3 lo0 - -
192.168.117.208 127.0.0.1 UGHS 0 3 lo0 - -
192.168.117.255 192.168.117.207 UHSb 0 0 et0 - - =>
192.168.117.255 192.168.117.208 UHSb 0 0 et1 - - =>
192.168.117.255 192.168.117.206 UHSb 0 4 en2 - -
192.168.118.0 192.168.118.206 UHSb 0 0 en0 - - =>
192.168.118/24 192.168.118.206 U 3 5159085 en0 - -
192.168.118.206 127.0.0.1 UGHS 0 698401 lo0 - -
192.168.118.255 192.168.118.206 UHSb 0 3448 en0 - -
192.168.120.0 192.168.120.206 UHSb 0 0 en2 - - =>
192.168.120/24 192.168.120.206 U 1 4208483 en2 - -
192.168.120.206 127.0.0.1 UGHS 0 1240676 lo0 - -
192.168.120.255 192.168.120.206 UHSb 0 4220 en2 - -
192.168.121.0 192.168.121.206 UHSb 0 0 en1 - - =>
192.168.121/24 192.168.121.206 U 0 0 en1 - -
192.168.121.206 127.0.0.1 UGHS 0 12907 lo0 - -
192.168.121.255 192.168.121.206 UHSb 0 8 en1 - -
192.168.123.0 192.168.123.206 UHSb 0 0 en3 - - =>
192.168.123/24 192.168.123.206 U 2 4504851 en3 - -
192.168.123.206 127.0.0.1 UGHS 7 614503 lo0 - -
192.168.123.255 192.168.123.206 UHSb 0 4072 en3 - -

Route Tree for Protocol Family 24 (Internet v6):
::1 ::1 UH 0 480 lo0 - -
# hostname
pr2serv
#


**********************
# netstat -rn
Routing tables
Destination Gateway Flags Refs Use If Exp Groups

Route Tree for Protocol Family 2 (Internet):
127/8 127.0.0.1 U 42 8865478 lo0 - -
192.168.10/24 192.168.117.252 UGS 4 57161 en0 - -
192.168.117.0 192.168.117.211 UHSb 0 0 en2 - - =>
192.168.117/24 192.168.117.211 U 46 7221620 en2 - -
192.168.117.211 127.0.0.1 UGHS 57 2380091 lo0 - -
192.168.117.255 192.168.117.211 UHSb 0 4 en2 - -
192.168.118.0 192.168.118.211 UHSb 0 0 en0 - - =>
192.168.118/24 192.168.118.211 U 2 4258920 en0 - -
192.168.118.211 127.0.0.1 UGHS 0 439505 lo0 - -
192.168.118.255 192.168.118.211 UHSb 0 9 en0 - -
192.168.120.0 192.168.120.211 UHSb 0 0 en2 - - =>
192.168.120/24 192.168.120.211 U 1 3383026 en2 - -
192.168.120.211 127.0.0.1 UGHS 0 435761 lo0 - -
192.168.120.255 192.168.120.211 UHSb 0 9 en2 - -
192.168.121.0 192.168.121.211 UHSb 0 0 en1 - - =>
192.168.121/24 192.168.121.211 U 0 0 en1 - -
192.168.121.211 127.0.0.1 UGHS 0 63 lo0 - -
192.168.121.255 192.168.121.211 UHSb 0 8 en1 - -
192.168.123.0 192.168.123.211 UHSb 0 0 en3 - - =>
192.168.123/24 192.168.123.211 U 1 3675762 en3 - -
192.168.123.211 127.0.0.1 UGHS 0 851044 lo0 - -
192.168.123.255 192.168.123.211 UHSb 0 9 en3 - -

Route Tree for Protocol Family 24 (Internet v6):
::1 ::1 UH 0 544 lo0 - -
# hostname
px1serv
#

Sorry, you cannot face any problem with NFS in your Cluster because you did neither configure any directory for import nor for export. At least I cannot see any directory here. Is that clshowres output really complete?

**** NFS exports in a resource group *****
NFS filesystem that is managed by HACMP.

* On each node verify that NFS is running or not
# lssrc –g nfs (after HACMP started)

* Modify the resource group R_G_A  /fsA to NFS export and set the option: Filesystems mounted before IP Configured to TRUE.

* smitty hacmp  Extended Resource Configuration  HACMP extended resource group configuration  change show Resources and Attributes for a Resource Group  R_G_A
+ Set Filesystems mounted before IP Configured to true
+ Set Filesystems/Directories to Export (NFSv2/3) to /fsA

* Synchronize the resources  Extended Configuration  Extended Verification and Synchronization

* more /var/hacmp/adm/cluster.log

* # lsnfsexp

* # cat /etc/xtab ( Note /etc/exports is not used by HACMP )
# cat /etc/exports

* Log in on Node_3 as root

* Create a directory for testing the NFS mount
# mkdir /node1
# mount Node_1#:/fsA /node1
( If this fails, check your /etc/hosts file – remember that node3 address must be resolvable in all the /etc/hosts files
# mount
# df –g


Modifying the NFS export Options (override file if you want to use non-root users to write on NFS mounted filesystem)

* on Node_2  #lsnfsexp

* # lsnfsexp > /usr/es/sbin/cluster/etc/exports

* Edit /usr/es/sbin/cluster/etc/exports and add the clients to the list of hosts for /fsA

* Copy /usr/es/sbin/cluster/etc/exports to Node_1

* Restart Cluster Services on Node_1

* Node_3 # touch /node1/file.txt


**** NFS cross-mount within the cluster ****
Nodes in the cluster act as both client and server ; Same name mount points

* On both nodes create a directory /samefs

* On Node_1 edit R_G_A resource group so that /fsA NFS filesystem will be NFS mounted on the /samefs mount point fo all nodes in that resource group

# smitty hacmp  Extended Configuration  Extended Resource Configuration  HACMP Extended Resource Group Configuration  Change/Show Resources and Attributes for a Resource Group  Select R_G_A  Set Filesystems/Directories to NFS mount: /samefs;/fsA

* Synchronize the resources  Extended Configuration  Extended Verification and Synchronization

# mount
# smitty clstop ( Node_1 ; so that Node_2 should have all the resource groups and that NFS mounts are OK)
# clRGinfo
# df -g