Hi
I am having two node cluster. I was trying to change the to use scsi3 reservation for all disks having two paths. Please let me know where I went wrong or there any issue to implement the same.
On node1
Code :
bash-3.2# cldev status
=== Cluster DID Devices ===
Device Instance Node Status
--------------- ---- ------
/dev/did/rdsk/d1 node2 Ok
/dev/did/rdsk/d10 node1 Ok
node2 Ok
/dev/did/rdsk/d11 node2 Ok
/dev/did/rdsk/d12 node1 Ok
/dev/did/rdsk/d14 node1 Ok
/dev/did/rdsk/d3 node2 Ok
/dev/did/rdsk/d4 node1 Ok
node2 Ok
/dev/did/rdsk/d5 node1 Ok
node2 Ok
/dev/did/rdsk/d6 node1 Ok
node2 Ok
/dev/did/rdsk/d7 node1 Ok
node2 Ok
/dev/did/rdsk/d8 node2 Ok
/dev/did/rdsk/d9 node1 Ok
node2 Ok
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2# clq status
=== Cluster Quorum ===
--- Quorum Votes Summary from latest node reconfiguration ---
Needed Present Possible
------ ------- --------
2 3 3
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status
--------- ------- -------- ------
node2 1 1 Online
node1 1 1 Online
--- Quorum Votes by Device (current status) ---
Device Name Present Possible Status
----------- ------- -------- ------
d6 1 1 Online
bash-3.2#
bash-3.2# cldev list -v
DID Device Full Device Path
---------- ----------------
d1 node2:/dev/rdsk/c0d0
d2 node2:/dev/rdsk/c1t0d0
d3 node2:/dev/rdsk/c0d1
d4 node2:/dev/rdsk/c2t600144F055753E0900080027FCD22200d0
d4 node1:/dev/rdsk/c2t600144F055753E0900080027FCD22200d0
d5 node2:/dev/rdsk/c2t600144F055753E0100080027FCD22200d0
d5 node1:/dev/rdsk/c2t600144F055753E0100080027FCD22200d0
d6 node2:/dev/rdsk/c2t600144F055753DD800080027FCD22200d0
d6 node1:/dev/rdsk/c2t600144F055753DD800080027FCD22200d0
d7 node2:/dev/rdsk/c2t600144F055753DD100080027FCD22200d0
d7 node1:/dev/rdsk/c2t600144F055753DD100080027FCD22200d0
d8 node2:/dev/rdsk/c3t0d0
d9 node2:/dev/rdsk/c2t600144F055753DE500080027FCD22200d0
d9 node1:/dev/rdsk/c2t600144F055753DE500080027FCD22200d0
d10 node2:/dev/rdsk/c2t600144F055753DED00080027FCD22200d0
d10 node1:/dev/rdsk/c2t600144F055753DED00080027FCD22200d0
d11 node2:/dev/rdsk/c3t1d0
d12 node1:/dev/rdsk/c0d0
d13 node1:/dev/rdsk/c1t0d0
d14 node1:/dev/rdsk/c0d1
bash-3.2# metaset
Set name = test-set, Set number = 1
Host Owner
node1 Yes
node2
Driv Dbase
d7 Yes
d9 Yes
bash-3.2#
bash-3.2#
bash-3.2# clq add d4
bash-3.2#
bash-3.2# clq status
=== Cluster Quorum ===
--- Quorum Votes Summary from latest node reconfiguration ---
Needed Present Possible
------ ------- --------
3 4 4
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status
--------- ------- -------- ------
node2 1 1 Online
node1 1 1 Online
--- Quorum Votes by Device (current status) ---
Device Name Present Possible Status
----------- ------- -------- ------
d6 1 1 Online
d4 1 1 Online
bash-3.2#
bash-3.2#
bash-3.2# clq remove d4
bash-3.2#
bash-3.2# clq status
=== Cluster Quorum ===
--- Quorum Votes Summary from latest node reconfiguration ---
Needed Present Possible
------ ------- --------
2 3 3
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status
--------- ------- -------- ------
node2 1 1 Online
node1 1 1 Online
--- Quorum Votes by Device (current status) ---
Device Name Present Possible Status
----------- ------- -------- ------
d6 1 1 Online
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2# cluster show|grep global_fencing
global_fencing: pathcount
bash-3.2#
bash-3.2# cldev show |grep default_fencing
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
bash-3.2#
bash-3.2# cldev show d6 |grep default_fencing
default_fencing: his
bash-3.2#
bash-3.2#
bash-3.2# cluster set -p global_fencing=prefer3
Warning: Device instance d6 is a quorum device - fencing protocol remains PATHCOUNT for the device.
Warning: Device instance d8 supports SCSI-2 only and set to use pathcount fencing protocol.
Warning: Device instance d11 supports SCSI-2 only and set to use pathcount fencing protocol.
Updating shared devices on node 1
Updating shared devices on node 2
bash-3.2#
bash-3.2#
bash-3.2# cluster show|grep global_fencing
global_fencing: prefer3
bash-3.2#
bash-3.2#
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: global
default_fencing: pathcount
default_fencing: global
default_fencing: pathcount
default_fencing: global
default_fencing: global
default_fencing: pathcount
default_fencing: global
default_fencing: global
default_fencing: global
bash-3.2#
bash-3.2#
bash-3.2# cldev show d6 |grep default_fencing
default_fencing: pathcount
bash-3.2# cldev show d8 |grep default_fencing
default_fencing: pathcount
bash-3.2# cldev show d11 |grep default_fencing
default_fencing: pathcount
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2# clq show d6
=== Quorum Devices ===
Quorum Device Name: d6
Enabled: yes
Votes: 1
Global Name: /dev/did/rdsk/d6s2
Type: shared_disk
Access Mode: scsi2
Hosts (enabled): node2, node1
bash-3.2#
bash-3.2#
bash-3.2# clq add d4
^C
^C
^C
Now above command hanged & no # prompt was coming on the terminal.
In above process, I want to follow process where I was trying to add first another quorum d4 & then remove the d6 quorum & then change the default_fencing properties of d6 & then add the d6 as quorum & then remove the d4 as quorum. Moreover I am not concern about d8 & d11 as those are local disk . After above process scsi3 reservation was implemented properly.
Need to abruptly shutdown both the nodes as even cluster shutdown comman was not working.
While rebooting both the nodes as nothing was working after running last command, I am getting following the messages on both nodes.
NOTICE: CMM: Cluster Doesn't have operational quorum yet; waiting for quorum
Then I used the following procedure on both nodes to recover my cluster where I deleted quorum device d4 entries from infrastructure file & reboot the nodes. After rebooting the nodes in cluster I wanted to follow another procedure where I will put the cluster in installed mode & remove d6 from quorum device & change it default_fencing properties to global & then add d6 as quorum device.
On node1
Code :
login as: root
Using keyboard-interactive authentication.
Password:
Last login: Fri Sep 4 15:06:58 2015 from 192.168.11.85
Oracle Corporation SunOS 5.10 Generic Patch January 2005
# bash
bash-3.2#
bash-3.2# cp /etc/cl
clri cluster/
bash-3.2# cp /etc/cl
clri cluster/
bash-3.2# cp /etc/cluster/ccr/global/infrastructure /etc/cluster/ccr/global/infr astructure-04092015
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2# ls -l /etc/cluster/ccr/global/infrastructure*
-rw------- 1 root root 5527 Sep 4 14:52 /etc/cluster/ccr/global/infrastructure
-rw------- 1 root root 5527 Sep 4 23:53 /etc/cluster/ccr/global/infrastructure-04092015
-rw------- 1 root root 5526 Sep 4 14:52 /etc/cluster/ccr/global/infrastructure.bak
-rw-r--r-- 1 root root 0 Sep 4 23:38 /etc/cluster/ccr/global/infrastructure.new
bash-3.2#
bash-3.2# grep access_mode /etc/cluster/ccr/global/infrastructure
cluster.quorum_devices.1.properties.access_mode scsi2
cluster.quorum_devices.2.properties.access_mode scsi3
bash-3.2#
bash-3.2# hostname
node1
bash-3.2# cat /etc/cluster/nodeid
2
bash-3.2#
bash-3.2# cat /etc/cluster/ccr/global/infrastructure |grep quorum_devices
cluster.quorum_devices.1.name d6
cluster.quorum_devices.1.state enabled
cluster.quorum_devices.1.properties.votecount 1
cluster.quorum_devices.1.properties.gdevname /dev/did/rdsk/d6s2
cluster.quorum_devices.1.properties.path_1 enabled
cluster.quorum_devices.1.properties.path_2 enabled
cluster.quorum_devices.1.properties.access_mode scsi2
cluster.quorum_devices.1.properties.type shared_disk
cluster.quorum_devices.2.name d4
cluster.quorum_devices.2.state enabled
cluster.quorum_devices.2.properties.votecount 1
cluster.quorum_devices.2.properties.gdevname /dev/did/rdsk/d4s2
cluster.quorum_devices.2.properties.path_1 enabled
cluster.quorum_devices.2.properties.path_2 enabled
cluster.quorum_devices.2.properties.access_mode scsi3
cluster.quorum_devices.2.properties.type shared_disk
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2# vi /etc/cluster/ccr/global/infrastructure
"/etc/cluster/ccr/global/infrastructure" 111 lines, 5527 characters
ccr_gennum 10
ccr_checksum 07D66A5E8801E180DA751C80AFA57DA6
cluster.name SunCluster
cluster.state enabled
cluster.properties.cluster_id 0x55794F42
cluster.properties.installmode disabled
cluster.properties.private_net_number 10.240.0.0
cluster.properties.cluster_netmask 255.255.255.0
cluster.properties.private_netmask 255.255.255.128
cluster.properties.private_subnet_netmask 255.255.255.248
cluster.properties.private_user_net_number 10.240.0.64
cluster.properties.private_user_netmask 255.255.255.224
cluster.properties.private_maxnodes 6
cluster.properties.private_maxprivnets 10
cluster.properties.zoneclusters 12
cluster.properties.auth_joinlist_type sys
cluster.properties.auth_joinlist_hostslist .
cluster.properties.transport_heartbeat_timeout 10000
cluster.properties.transport_heartbeat_quantum 1000
cluster.properties.udp_session_timeout 480
cluster.properties.cmm_version 1
cluster.nodes.1.name node2
cluster.nodes.1.state enabled
cluster.nodes.1.properties.private_hostname clusternode1-priv
cluster.nodes.1.properties.quorum_vote 1
cluster.nodes.1.properties.quorum_resv_key 0x55794F4200000001
cluster.nodes.1.adapters.1.name e1000g2
cluster.nodes.1.adapters.1.state enabled
cluster.nodes.1.adapters.1.properties.device_name e1000g
cluster.nodes.1.adapters.1.properties.device_instance 2
cluster.nodes.1.adapters.1.properties.transport_type dlpi
cluster.nodes.1.adapters.1.properties.lazy_free 1
cluster.nodes.1.adapters.1.properties.dlpi_heartbeat_timeout 10000
cluster.nodes.1.adapters.1.properties.dlpi_heartbeat_quantum 1000
cluster.nodes.1.adapters.1.properties.nw_bandwidth 80
cluster.nodes.1.adapters.1.properties.bandwidth 70
cluster.nodes.1.adapters.1.properties.ip_address 10.240.0.17
cluster.nodes.1.adapters.1.properties.netmask 255.255.255.248
cluster.nodes.1.adapters.1.ports.1.name 0
cluster.nodes.1.adapters.1.ports.1.state enabled
cluster.nodes.1.adapters.2.name e1000g3
cluster.nodes.1.adapters.2.state enabled
cluster.nodes.1.adapters.2.properties.device_name e1000g
/quorum_devices
cluster.nodes.2.adapters.2.properties.device_name e1000g
cluster.nodes.2.adapters.2.properties.device_instance 3
cluster.nodes.2.adapters.2.properties.transport_type dlpi
cluster.nodes.2.adapters.2.properties.lazy_free 1
cluster.nodes.2.adapters.2.properties.dlpi_heartbeat_timeout 10000
cluster.nodes.2.adapters.2.properties.dlpi_heartbeat_quantum 1000
cluster.nodes.2.adapters.2.properties.nw_bandwidth 80
cluster.nodes.2.adapters.2.properties.bandwidth 70
cluster.nodes.2.adapters.2.properties.ip_address 10.240.0.10
cluster.nodes.2.adapters.2.properties.netmask 255.255.255.248
cluster.nodes.2.adapters.2.state enabled
cluster.nodes.2.adapters.2.ports.1.name 0
cluster.nodes.2.adapters.2.ports.1.state enabled
cluster.nodes.2.cmm_version 1
cluster.cables.1.properties.end1 cluster.nodes.2.adapters.1.ports.1
cluster.cables.1.properties.end2 cluster.nodes.1.adapters.1.ports.1
cluster.cables.1.state enabled
cluster.cables.2.properties.end1 cluster.nodes.2.adapters.2.ports.1
cluster.cables.2.properties.end2 cluster.nodes.1.adapters.2.ports.1
cluster.cables.2.state enabled
cluster.quorum_devices.1.name d6
cluster.quorum_devices.1.state enabled
cluster.quorum_devices.1.properties.votecount 1
cluster.quorum_devices.1.properties.gdevname /dev/did/rdsk/d6s2
cluster.quorum_devices.1.properties.path_1 enabled
cluster.quorum_devices.1.properties.path_2 enabled
cluster.quorum_devices.1.properties.access_mode scsi2
cluster.quorum_devices.1.properties.type shared_disk
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
"/etc/cluster/ccr/global/infrastructure" 103 lines, 5134 characters
bash-3.2#
bash-3.2#
bash-3.2# cat /etc/cluster/ccr/global/infrastructure |grep quorum_devices
cluster.quorum_devices.1.name d6
cluster.quorum_devices.1.state enabled
cluster.quorum_devices.1.properties.votecount 1
cluster.quorum_devices.1.properties.gdevname /dev/did/rdsk/d6s2
cluster.quorum_devices.1.properties.path_1 enabled
cluster.quorum_devices.1.properties.path_2 enabled
cluster.quorum_devices.1.properties.access_mode scsi2
cluster.quorum_devices.1.properties.type shared_disk
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2# /usr/cluster/lib/sc/ccradm recover -o infrastructure
Invalid ccr table file infrastructure (No such file or directory)
bash-3.2#
bash-3.2#
bash-3.2# /usr/cluster/lib/sc/ccradm recover -o /etc/cluster/ccr/global/infrastructure
bash-3.2#
bash-3.2# reboot -- -r
login as: root
Using keyboard-interactive authentication.
Password:
Last login: Fri Sep 4 23:49:55 2015 from 192.168.11.85
Oracle Corporation SunOS 5.10 Generic Patch January 2005
# bash
bash-3.2#
bash-3.2#
bash-3.2# Sep 5 00:08:24 node1 sendmail[826]: [ID 702911 mail.alert] unable to qualify my own domain name (node1) -- using short name
Sep 5 00:08:24 node1 sendmail[819]: [ID 702911 mail.alert] unable to qualify my own domain name (node1) -- using short name
bash-3.2# cldev status
=== Cluster DID Devices ===
Device Instance Node Status
--------------- ---- ------
/dev/did/rdsk/d1 node2 Ok
/dev/did/rdsk/d10 node1 Ok
node2 Ok
/dev/did/rdsk/d11 node2 Ok
/dev/did/rdsk/d12 node1 Ok
/dev/did/rdsk/d14 node1 Ok
/dev/did/rdsk/d3 node2 Ok
/dev/did/rdsk/d4 node1 Ok
node2 Ok
/dev/did/rdsk/d5 node1 Ok
node2 Ok
/dev/did/rdsk/d6 node1 Ok
node2 Ok
/dev/did/rdsk/d7 node1 Ok
node2 Ok
/dev/did/rdsk/d8 node2 Ok
/dev/did/rdsk/d9 node1 Ok
node2 Ok
bash-3.2#
bash-3.2#
bash-3.2# clq status
=== Cluster Quorum ===
--- Quorum Votes Summary from latest node reconfiguration ---
Needed Present Possible
------ ------- --------
2 2 3
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status
--------- ------- -------- ------
node2 1 1 Online
node1 1 1 Online
--- Quorum Votes by Device (current status) ---
Device Name Present Possible Status
----------- ------- -------- ------
d6 1 1 Online
bash-3.2#
bash-3.2#
bash-3.2#
bash-3.2# cluster show|grep global_fencing
global_fencing: prefer3
bash-3.2#
bash-3.2#
bash-3.2# cldev show d6 |grep default_fencing
default_fencing: pathcount
bash-3.2#
bash-3.2# cldev show d4 |grep default_fencing
default_fencing: global
bash-3.2#
bash-3.2# clq show d4
clq: (C952776) Device "d4" does not exist.
bash-3.2# clq show d6
=== Quorum Devices ===
Quorum Device Name: d6
Enabled: yes
Votes: 1
Global Name: /dev/did/rdsk/d6s2
Type: shared_disk
Access Mode: scsi2
Hosts (enabled): node2, node1
bash-3.2#
bash-3.2#
bash-3.2# clq remove d6
clq: (C115344) Cluster quorum could be compromised if you remove "d6".
bash-3.2#
bash-3.2# cluster set -p installmode=enabled
bash-3.2#
bash-3.2# clq remove d6
bash-3.2#
bash-3.2# clq status
=== Cluster Quorum ===
--- Quorum Votes Summary from latest node reconfiguration ---
Needed Present Possible
------ ------- --------
2 2 2
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status
--------- ------- -------- ------
node2 1 1 Online
node1 1 1 Online
bash-3.2#
bash-3.2# cldev show d6 |grep default_fencing
default_fencing: pathcount
bash-3.2#
bash-3.2# cldev set -p default_fencing=global d6
Updating shared devices on node 1
Updating shared devices on node 2
bash-3.2#
bash-3.2# cldev show d6 |grep default_fencing
default_fencing: global
bash-3.2#
bash-3.2#
bash-3.2# clq add d6
Again after running last command that got hanged & didn't got any # prompt.
I need to shutdown abruptly both the nodes again.
Kindly let me know how can I make d6(quorum device) as scsi3 reservation