Login or Register to Ask a Question and Join Our Community


Solaris

sc 3.2, load balanced apache offline on both nodes

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Solaris sc 3.2, load balanced apache offline on both nodes
# 1  
Old 07-17-2009
Question sc 3.2, load balanced apache offline on both nodes
Hi
I am trying to set up apache on the only 2 nodes of sun cluster 3.2. nodes are running solaris 10-x86

For some reasons, apache resource on both nodes goes offline in some seconds.
Quote:
-bash-3.00# clrs status

=== Cluster Resources ===

Resource Name Node Name State Status Message
------------- --------- ----- --------------
apsshared-sa tommy_sun1 Offline Offline
tommy_sun Online Online - SharedAddress online.

apache-rs tommy_sun1 Offline Offline - Successfully stopped Apache Web Server.
tommy_sun Offline Offline - Successfully stopped Apache Web Server.
There is a shared address resource which is online

rs, rg list and show below,

Quote:
-bash-3.00# clrg list
apshared-rg
apache-rg
-bash-3.00# clrs list
apsshared-sa
apache-rs
Quote:
-bash-3.00# clrs show

=== Resources ===

Resource: apsshared-sa
Type: SUNW.SharedAddress:2
Type_version: 2
Group: apshared-rg
R_description:
Resource_project_name: default
Enabled{tommy_sun1}: True
Enabled{tommy_sun}: True
Monitored{tommy_sun1}: True
Monitored{tommy_sun}: True

Resource: apache-rs
Type: SUNW.apache:4.1
Type_version: 4.1
Group: apache-rg
R_description:
Resource_project_name: default
Enabled{tommy_sun1}: True
Enabled{tommy_sun}: True
Monitored{tommy_sun1}: True
Monitored{tommy_sun}: True
Quote:
-bash-3.00# clrg show

=== Resource Groups and Resources ===

Resource Group: apshared-rg
RG_description: <NULL>
RG_mode: Failover
RG_state: Managed
Failback: False
Nodelist: tommy_sun1 tommy_sun

--- Resources for Group apshared-rg ---

Resource: apsshared-sa
Type: SUNW.SharedAddress:2
Type_version: 2
Group: apshared-rg
R_description:
Resource_project_name: default
Enabled{tommy_sun1}: True
Enabled{tommy_sun}: True
Monitored{tommy_sun1}: True
Monitored{tommy_sun}: True

Resource Group: apache-rg
RG_description: <NULL>
RG_mode: Scalable
RG_state: Managed
Failback: False
Nodelist: tommy_sun1 tommy_sun

--- Resources for Group apache-rg ---

Resource: apache-rs
Type: SUNW.apache:4.1
Type_version: 4.1
Group: apache-rg
R_description:
Resource_project_name: default
Enabled{tommy_sun1}: True
Enabled{tommy_sun}: True
Monitored{tommy_sun1}: True
Monitored{tommy_sun}: True
clrg online apache-rg
Quote:
clrg: (C748634) Resource group apache-rg failed to start on chosen node and might fail over to other node(s)
clrg: (C135343) No primary node could be found for resource group apache-rg; it remains offline
[/QUOTE]

dmesg
Quote:
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_PENDING_ONLINE
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_PENDING_ONLINE
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STARTING_DEPEND
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_OK_TO_START
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STARTING
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_svc_start (scalable svc)> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/sc/ssm_wrapper>:tag=<apache-rg.apache-rs.0>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STARTING_DEPEND
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_OK_TO_START
Jul 17 09:04:48 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STARTING
Jul 17 09:04:49 tommy_sun SC[SUNW.apache:4.1,apache-rg,apache-rs,SSM_START]: [ID 487022 daemon.notice] The networking components for scalable resource apache-rs have been configured successfully for method START.
Jul 17 09:04:49 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 708422 daemon.error] Command {/usr/apache2/bin/apachectl configtest >/dev/null 2>&1} failed: httpd cannot parse httpd.conf.
Jul 17 09:04:49 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 251552 daemon.error] Failed to validate configuration.
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_START_FAILED
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_FAULTED
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_PENDING_OFF_START_FAILED
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_UNKNOWN
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Stopping>
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STOPPING
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 938318 daemon.error] Method <apache_svc_start (scalable svc)> failed on resource <apache-rs> in resource group <apache-rg> [exit code <1>, time used: 0% of timeout <300 seconds>]
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_START_FAILED
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_PENDING_OFF_START_FAILED
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_FAULTED
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STOPPING
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_UNKNOWN
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Stopping>
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_svc_stop> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 09:04:49 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscapc/bin/apache_svc_stop>:tag=<apache-rg.apache-rs.1>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 09:04:50 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 584207 daemon.notice] Stopping Apache Web Server.
Jul 17 09:04:50 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 401400 daemon.notice] Successfully stopped the application
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_OFFLINE
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Successfully stopped Apache Web Server.>
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <apache_svc_stop> completed successfully for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, time used: 0% of timeout <300 seconds>
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_OFFLINE
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_OFFLINE_START_FAILED
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_OFFLINE
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 447451 daemon.notice] Not attempting to start resource group <apache-rg> on node <tommy_sun> because this resource group has already failed to start on this node 2 or more times in the past 3600 seconds
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_OFFLINE
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Successfully stopped Apache Web Server.>
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_OFFLINE
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_OFFLINE_START_FAILED
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_OFFLINE
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 447451 daemon.notice] Not attempting to start resource group <apache-rg> on node <tommy_sun1> because this resource group has already failed to start on this node 2 or more times in the past 3600 seconds
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 447451 daemon.notice] Not attempting to start resource group <apache-rg> on node <tommy_sun> because this resource group has already failed to start on this node 2 or more times in the past 3600 seconds
Jul 17 09:04:50 tommy_sun Cluster.RGM.global.rgmd: [ID 674214 daemon.notice] rebalance: no primary node is currently found for resource group <apache-rg>.
Can someone please help me?
Thanks in advance.
Smilie
# 2  
Old 07-17-2009
looks like your apache is misconfigured...

Code:
Jul 17 09:04:49 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 708422 daemon.error] Command {/usr/apache2/bin/apachectl configtest >/dev/null 2>&1} failed: httpd cannot parse httpd.conf.
Jul 17 09:04:49 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 251552 daemon.error] Failed to validate configuration.

# 3  
Old 07-17-2009
Quote:
Originally Posted by DukeNuke2
looks like your apache is misconfigured...

Code:
Jul 17 09:04:49 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 708422 daemon.error] Command {/usr/apache2/bin/apachectl configtest >/dev/null 2>&1} failed: httpd cannot parse httpd.conf.
Jul 17 09:04:49 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 251552 daemon.error] Failed to validate configuration.

Thank you for quick reponse. Yes you are right, I disabled sun cluster now and tried manually starting apache2 with svcadm.
I had deleted "Listen 80" in http.conf by mistake.(actuall apache was working earlier but as it was not working with sun cluster, I had changed Listen directive to IP 80 and recently I disabled IP 80 but forgot to add Listen 80 again)

apache2 starts fine on one node now. Other node shows unitialized state for apache2? what could be the issue?

I am sorry but this is taking to the basics but I think there will be problems when I try apache2 with sc.
Quote:
svcs -l apache2
fmri svc:/network/http:apache2
name Apache 2 HTTP server
enabled false
state uninitialized
next_state none
state_time Fri Jul 17 09:07:45 2009
restarter svc:/system/cluster/sc_restarter:default
dependency require_all/error svc:/milestone/network:default (online)
dependency require_all/none svc:/system/filesystem/local:default (online)
dependency optional_all/error svc:/system/filesystem/autofs:default (disabled)
I am not able to see fresh logs in apache2/error_logs

Earlier it was working fine before sun cluster config.

---------- Post updated at 10:17 AM ---------- Previous update was at 10:01 AM ----------

I don't understand why is restarter is sc_restarter (restarter svc:/system/cluster/sc_restarter:default) instead of
svc:/system/svc/restarter:default Smilie

---------- Post updated at 10:29 AM ---------- Previous update was at 10:17 AM ----------

svccfg apache2
delprop general/restarter
reboot

apache2 is now online other node as well Smilie Now I will enable /online rs and rgs
# 4  
Old 07-17-2009
clrg online apache-rg
clrg: (C748634) Resource group apache-rg failed to start on chosen node and might fail over to other node(s)
clrg: (C135343) No primary node could be found for resource group apache-rg; it remains offline

Quote:
Fri Jul 17 10:22:04 CDT 2009
Jul 17 10:21:23 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_MON_STARTING
Jul 17 10:21:23 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_PENDING_ON_STARTED
Jul 17 10:21:23 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_monitor_start (scalable svc)> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 10:21:23 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/sc/ssm_wrapper>:tag=<apache-rg.apache-rs.7>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 10:21:23 tommy_sun SC[SUNW.apache:4.1,apache-rg,apache-rs,SSM_MONITOR_START]: [ID 487022 daemon.notice] The networking components for scalable resource apache-rs have been configured successfully for method MONITOR_START.
Jul 17 10:21:23 tommy_sun Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <apache_monitor_start (scalable svc)> completed successfully for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, time used: 0% of timeout <300 seconds>
Jul 17 10:21:23 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_ONLINE
Jul 17 10:21:23 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_ONLINE
Jul 17 10:21:23 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_JUST_STARTED
Jul 17 10:21:23 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_ONLINE_UNMON
Jul 17 10:21:24 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_ONLINE
Jul 17 10:21:24 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_ONLINE
Jul 17 10:21:25 tommy_sun Cluster.PMF.pmfd: [ID 887656 daemon.notice] Process: tag="apache-rg,apache-rs,0.svc", cmd="/bin/sh -c /usr/apache2/bin/apachectl start", Failed to stay up.
Jul 17 10:21:25 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_probe]: [ID 670283 daemon.notice] Issuing a resource restart request because the application exited.
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 494478 daemon.notice] resource apache-rs in resource group apache-rg has requested restart of the resource on tommy_sun.
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_ON_PENDING_R_RESTART
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_FAULTED
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Service daemon not running.>
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_MON_STOPPING
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_monitor_stop (scalable svc)> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/sc/ssm_wrapper>:tag=<apache-rg.apache-rs.8>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_FAULTED
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Service daemon not running.>
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_ON_PENDING_R_RESTART
Jul 17 10:21:25 tommy_sun SC[SUNW.apache:4.1,apache-rg,apache-rs,SSM_MONITOR_STOP]: [ID 487022 daemon.notice] The networking components for scalable resource apache-rs have been configured successfully for method MONITOR_STOP.
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <apache_monitor_stop (scalable svc)> completed successfully for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, time used: 0% of timeout <300 seconds>
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_ONLINE_UNMON
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STOPPING
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_svc_stop> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_UNKNOWN
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Stopping>
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscapc/bin/apache_svc_stop>:tag=<apache-rg.apache-rs.1>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 10:21:25 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 584207 daemon.notice] Stopping Apache Web Server.
Jul 17 10:21:25 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 401400 daemon.notice] Successfully stopped the application
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_OFFLINE
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Successfully stopped Apache Web Server.>
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <apache_svc_stop> completed successfully for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, time used: 0% of timeout <300 seconds>
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_OFFLINE
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STARTING_DEPEND
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_OK_TO_START
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_PENDING_ONLINE
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STARTING
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_svc_start (scalable svc)> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 10:21:25 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/sc/ssm_wrapper>:tag=<apache-rg.apache-rs.0>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_ONLINE_UNMON
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_UNKNOWN
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Stopping>
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STOPPING
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_OFFLINE
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Successfully stopped Apache Web Server.>
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_OFFLINE
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STARTING_DEPEND
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_OK_TO_START
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_PENDING_ONLINE
Jul 17 10:21:26 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STARTING
Jul 17 10:21:27 tommy_sun SC[SUNW.apache:4.1,apache-rg,apache-rs,SSM_START]: [ID 487022 daemon.notice] The networking components for scalable resource apache-rs have been configured successfully for method START.
Jul 17 10:21:27 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 743923 daemon.notice] Starting server with command /usr/apache2/bin/apachectl start.
Jul 17 10:21:27 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 187883 daemon.error] Apache service with startup script </usr/apache2/bin/apachectl> does not configure PIDFILE.
Jul 17 10:21:27 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 569559 daemon.notice] Start of /usr/apache2/bin/apachectl start completed successfully.
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_ONLINE
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Completed successfully.>
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_ONLINE
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Completed successfully.>
Jul 17 10:21:27 tommy_sun Cluster.PMF.pmfd: [ID 887656 daemon.notice] Process: tag="apache-rg,apache-rs,0.svc", cmd="/bin/sh -c /usr/apache2/bin/apachectl start", Failed to stay up.
Jul 17 10:21:27 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 423137 daemon.error] A resource restart attempt on resource apache-rs in resource group apache-rg has been blocked because the number of restarts within the past Retry_interval (620 seconds) would exceed Retry_count (2)
Jul 17 10:21:27 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 819721 daemon.error] Failed to start Apache Web Server.
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 938318 daemon.error] Method <apache_svc_start (scalable svc)> failed on resource <apache-rs> in resource group <apache-rg> [exit code <1>, time used: 0% of timeout <300 seconds>]
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_START_FAILED
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_PENDING_OFF_START_FAILED
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_FAULTED
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STOPPING
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_UNKNOWN
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Stopping>
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_svc_stop> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 10:21:27 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscapc/bin/apache_svc_stop>:tag=<apache-rg.apache-rs.1>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 10:21:27 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 584207 daemon.notice] Stopping Apache Web Server.
Jul 17 10:21:27 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 401400 daemon.notice] Successfully stopped the application
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_OFFLINE
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Successfully stopped Apache Web Server.>
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <apache_svc_stop> completed successfully for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, time used: 0% of timeout <300 seconds>
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_OFFLINE
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_OFFLINE_START_FAILED
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_OFFLINE
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_PENDING_ONLINE
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STARTING_DEPEND
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_OK_TO_START
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STARTING
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_svc_start (scalable svc)> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/sc/ssm_wrapper>:tag=<apache-rg.apache-rs.0>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_START_FAILED
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_FAULTED
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_PENDING_OFF_START_FAILED
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_UNKNOWN
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Stopping>
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STOPPING
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_OFFLINE
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Successfully stopped Apache Web Server.>
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_OFFLINE
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_OFFLINE_START_FAILED
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_OFFLINE
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_PENDING_ONLINE
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STARTING_DEPEND
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_OK_TO_START
Jul 17 10:21:28 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STARTING
Jul 17 10:21:29 tommy_sun SC[SUNW.apache:4.1,apache-rg,apache-rs,SSM_START]: [ID 487022 daemon.notice] The networking components for scalable resource apache-rs have been configured successfully for method START.
Jul 17 10:21:29 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 743923 daemon.notice] Starting server with command /usr/apache2/bin/apachectl start.
Jul 17 10:21:29 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 187883 daemon.error] Apache service with startup script </usr/apache2/bin/apachectl> does not configure PIDFILE.
Jul 17 10:21:29 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 569559 daemon.notice] Start of /usr/apache2/bin/apachectl start completed successfully.
Jul 17 10:21:29 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_ONLINE
Jul 17 10:21:29 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Completed successfully.>
Jul 17 10:21:29 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_ONLINE
Jul 17 10:21:29 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Completed successfully.>
Jul 17 10:21:29 tommy_sun Cluster.PMF.pmfd: [ID 887656 daemon.notice] Process: tag="apache-rg,apache-rs,0.svc", cmd="/bin/sh -c /usr/apache2/bin/apachectl start", Failed to stay up.
Jul 17 10:21:30 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 423137 daemon.error] A resource restart attempt on resource apache-rs in resource group apache-rg has been blocked because the number of restarts within the past Retry_interval (620 seconds) would exceed Retry_count (2)
Jul 17 10:21:30 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 819721 daemon.error] Failed to start Apache Web Server.
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 938318 daemon.error] Method <apache_svc_start (scalable svc)> failed on resource <apache-rs> in resource group <apache-rg> [exit code <1>, time used: 0% of timeout <300 seconds>]
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_START_FAILED
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_PENDING_OFF_START_FAILED
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_FAULTED
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STOPPING
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_UNKNOWN
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Stopping>
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_svc_stop> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscapc/bin/apache_svc_stop>:tag=<apache-rg.apache-rs.1>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 10:21:30 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 584207 daemon.notice] Stopping Apache Web Server.
Jul 17 10:21:30 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 401400 daemon.notice] Successfully stopped the application
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_OFFLINE
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Successfully stopped Apache Web Server.>
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <apache_svc_stop> completed successfully for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, time used: 0% of timeout <300 seconds>
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_OFFLINE
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_OFFLINE_START_FAILED
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_OFFLINE
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 447451 daemon.notice] Not attempting to start resource group <apache-rg> on node <tommy_sun> because this resource group has already failed to start on this node 2 or more times in the past 3600 seconds
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_START_FAILED
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_FAULTED
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_PENDING_OFF_START_FAILED
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_UNKNOWN
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Stopping>
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STOPPING
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_OFFLINE
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Successfully stopped Apache Web Server.>
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_OFFLINE
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_OFFLINE_START_FAILED
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_OFFLINE
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 447451 daemon.notice] Not attempting to start resource group <apache-rg> on node <tommy_sun1> because this resource group has already failed to start on this node 2 or more times in the past 3600 seconds
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 447451 daemon.notice] Not attempting to start resource group <apache-rg> on node <tommy_sun> because this resource group has already failed to start on this node 2 or more times in the past 3600 seconds
Jul 17 10:21:30 tommy_sun Cluster.RGM.global.rgmd: [ID 674214 daemon.notice] rebalance: no primary node is currently found for resource group <apache-rg>.
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_PENDING_ONLINE
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_PENDING_ONLINE
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STARTING_DEPEND
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_OK_TO_START
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STARTING
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_svc_start (scalable svc)> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/sc/ssm_wrapper>:tag=<apache-rg.apache-rs.0>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STARTING_DEPEND
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_OK_TO_START
Jul 17 10:21:51 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STARTING
Jul 17 10:21:52 tommy_sun SC[SUNW.apache:4.1,apache-rg,apache-rs,SSM_START]: [ID 487022 daemon.notice] The networking components for scalable resource apache-rs have been configured successfully for method START.
Jul 17 10:21:52 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 743923 daemon.notice] Starting server with command /usr/apache2/bin/apachectl start.
Jul 17 10:21:52 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 187883 daemon.error] Apache service with startup script </usr/apache2/bin/apachectl> does not configure PIDFILE.
Jul 17 10:21:52 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 569559 daemon.notice] Start of /usr/apache2/bin/apachectl start completed successfully.
Jul 17 10:21:52 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_ONLINE
Jul 17 10:21:52 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Completed successfully.>
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_ONLINE
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Completed successfully.>
Jul 17 10:21:53 tommy_sun Cluster.PMF.pmfd: [ID 887656 daemon.notice] Process: tag="apache-rg,apache-rs,0.svc", cmd="/bin/sh -c /usr/apache2/bin/apachectl start", Failed to stay up.
Jul 17 10:21:53 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 423137 daemon.error] A resource restart attempt on resource apache-rs in resource group apache-rg has been blocked because the number of restarts within the past Retry_interval (620 seconds) would exceed Retry_count (2)
Jul 17 10:21:53 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_start]: [ID 819721 daemon.error] Failed to start Apache Web Server.
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 938318 daemon.error] Method <apache_svc_start (scalable svc)> failed on resource <apache-rs> in resource group <apache-rg> [exit code <1>, time used: 0% of timeout <300 seconds>]
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_START_FAILED
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_PENDING_OFF_START_FAILED
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_FAULTED
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_STOPPING
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_UNKNOWN
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Stopping>
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <apache_svc_stop> for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, timeout <300> seconds
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 703477 daemon.notice] 38 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscapc/bin/apache_svc_stop>:tag=<apache-rg.apache-rs.1>: Calling security_clnt_connect(..., host=<tommy_sun>, sec_type {0:WEAK, 1:STRONG, 2SmilieES} =<1>, ...)
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_START_FAILED
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_FAULTED
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_PENDING_OFF_START_FAILED
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_UNKNOWN
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Stopping>
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_STOPPING
Jul 17 10:21:53 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 584207 daemon.notice] Stopping Apache Web Server.
Jul 17 10:21:53 tommy_sun SC[,SUNW.apache:4.1,apache-rg,apache-rs,apache_svc_stop]: [ID 401400 daemon.notice] Successfully stopped the application
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun change to R_FM_OFFLINE
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun change to <Successfully stopped Apache Web Server.>
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <apache_svc_stop> completed successfully for resource <apache-rs>, resource group <apache-rg>, node <tommy_sun>, time used: 0% of timeout <300 seconds>
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun change to R_OFFLINE
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_OFFLINE_START_FAILED
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun change to RG_OFFLINE
Jul 17 10:21:53 tommy_sun Cluster.RGM.global.rgmd: [ID 447451 daemon.notice] Not attempting to start resource group <apache-rg> on node <tommy_sun> because this resource group has already failed to start on this node 2 or more times in the past 3600 seconds
Jul 17 10:21:54 tommy_sun Cluster.RGM.global.rgmd: [ID 784560 daemon.notice] resource apache-rs status on node tommy_sun1 change to R_FM_OFFLINE
Jul 17 10:21:54 tommy_sun Cluster.RGM.global.rgmd: [ID 922363 daemon.notice] resource apache-rs status msg on node tommy_sun1 change to <Successfully stopped Apache Web Server.>
Jul 17 10:21:54 tommy_sun Cluster.RGM.global.rgmd: [ID 443746 daemon.notice] resource apache-rs state on node tommy_sun1 change to R_OFFLINE
Jul 17 10:21:54 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_OFFLINE_START_FAILED
Jul 17 10:21:54 tommy_sun Cluster.RGM.global.rgmd: [ID 529407 daemon.notice] resource group apache-rg state on node tommy_sun1 change to RG_OFFLINE
Jul 17 10:21:54 tommy_sun Cluster.RGM.global.rgmd: [ID 447451 daemon.notice] Not attempting to start resource group <apache-rg> on node <tommy_sun1> because this resource group has already failed to start on this node 2 or more times in the past 3600 seconds
Jul 17 10:21:54 tommy_sun Cluster.RGM.global.rgmd: [ID 447451 daemon.notice] Not attempting to start resource group <apache-rg> on node <tommy_sun> because this resource group has already failed to start on this node 2 or more times in the past 3600 seconds
Jul 17 10:21:54 tommy_sun Cluster.RGM.global.rgmd: [ID 674214 daemon.notice] rebalance: no primary node is currently found for resource group <apache-rg>.


---------- Post updated at 10:41 AM ---------- Previous update was at 10:32 AM ----------

SOLVED

svcadm disable apache2 on both nodes
clrg online apache-rg

clrg status
clrg status

Quote:
=== Cluster Resource Groups ===

Group Name Node Name Suspended Status
---------- --------- --------- ------
apshared-rg tommy_sun1 No Online
tommy_sun No Offline

apache-rg tommy_sun1 No Online
tommy_sun No Online
so now it is working fine. I am thinking that after httpd.conf issues, it was 'restarter' for svc:apache2 on the other node was causing issues. Again I had to disable apache2 using svcadm and then sun cluster could online apache-rg. Also now I remember I was trying to get SUNW.SMF_proxy_scalable to get working with apache2 but for some reason that did not work so I was trying load balancing with sunw.apache.

Thank you for helping me out dukenuke2. appreciate it.Smilie
# 5  
Old 07-17-2009
Quote:
Originally Posted by upengan78
Thank you for helping me out dukenuke2. appreciate it.Smilie
most of the work was on your side Smilie. but thanks for the bits!
# 6  
Old 07-17-2009
Quote:
Originally Posted by DukeNuke2
most of the work was on your side Smilie. but thanks for the bits!
Smilie, I did not know where to start looking, you rightly pointed to httpd.conf and that's where the problem was Smilie

If you don't mind, can I mention here that I have a problem using SUNW.SMF_proxy_scalable DSA.

I am moving to SMF-proxy agent instead of load balancing I configured earlier. I am keeping the shared add rg as it is. deleted old apache-rg and started fresh.
Quote:
clrg create -p Maximum_primaries=2 -p Desired_primaries=2 -p RG_dependencies=apshared-rg -n tommy_sun,tommy_sun1 apache-rg
that works fine.

-bash-3.00# clresourcegroup show -v apache-rg
Quote:
=== Resource Groups and Resources ===

Resource Group: apache-rg
RG_description: <NULL>
RG_mode: Scalable
RG_state: Unmanaged
RG_project_name: default
RG_affinities: <NULL>
RG_SLM_type: manual
Auto_start_on_new_cluster: True
Failback: False
Nodelist: tommy_sun tommy_sun1
Maximum_primaries: 2
Desired_primaries: 2
RG_dependencies: apshared-rg
Implicit_network_dependencies: True
Global_resources_used: <All>
Pingpong_interval: 3600
Pathprefix: <NULL>
RG_System: False
Suspend_automatic_recovery: False
FYI, apshared-rg is shared add resource group.

Issue is this does not work,

clrs create -g apache-rg -t SUNW.Proxy_SMF_scalable -x Proxied_service_instances=/tmp/apa apache-rs apache-rs

Quote:
clrs: (C515685) : Invalid resource
clrs: (C891200) Failed to create resource "apache-rs"
-bash-3.00# cat /tmp/apa
Quote:
<svc:/network/http:apache2>,</var/svc/manifest/network/http-apache2.xml>
am I doing anything incorrect?
# 7  
Old 07-18-2009
Quote:
Originally Posted by upengan78
clrs create -g apache-rg -t SUNW.Proxy_SMF_scalable -x Proxied_service_instances=/tmp/apa apache-rs apache-rs
Hi,

why twice written apache-rs ?????
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Form balanced matrix by filtering data

I need to form a matrix out of unbalanced set of records. First eliminate the sample that do not have at least 3 variables (col2). So, in the example, samples 4 and 5 get eliminated. Then form a matrix of values (col3) from the samples using only variables that are present accross all samples.... (3 Replies)
Discussion started by: senhia83
3 Replies

2. Red Hat

How to troubleshoot a 1000 nodes Apache cluster?

Hi all. May I get some expert advice on troubleshooting performance issues of a 1000 nodes Apache LB cluster. Users report slow loading/response of webpages. Different websites are hosted on this cluster for different clients. But all are reporting the same issue. Could you please let me know... (1 Reply)
Discussion started by: admin_xor
1 Replies

3. Red Hat

apache high cpu load on high traffic

i have a Intel Quad Core Xeon X3440 (4 x 2.53GHz, 8MB Cache, Hyper Threaded) with 16gig and 1tb harddrive with a 1gb port and my apache is causing my cpu to go up to 100% on all four cores heres my http.config <IfModule prefork.c> StartServers 10 MinSpareServers 10 MaxSpareServers 15... (4 Replies)
Discussion started by: awww
4 Replies

4. Red Hat

Apache Load Balaning sticky session issue

Few doubts in apache load balancing - I want to clear my doubts about apache load balancing.i have configured apache load balancing for two application servers. 1. When we set load balancing for two application servers then for one connection to application server we will get one jsession id... (5 Replies)
Discussion started by: ganesh_gore
5 Replies

5. Red Hat

Conceptual details about apache load balancing

Hi Techies, Setup details - 1. H/W Load Balancer 2. Two Apache Servers with load balancing and rewrite rule 3. Two Application servers Request will be travel 1 then 2 then 3 Queries - 1. When any session is established I can see that requests of that session is going to both... (0 Replies)
Discussion started by: ganesh_gore
0 Replies

6. Web Development

Load Balancing in Apache

Hi All, I have one webserver which has an application for a set of internal users can be accessed by _http://server1.com I am planning to load balance this application. For that I have cloned this server and build a new one which can be accessed using _http://server2.com]Server2.com. Also i... (2 Replies)
Discussion started by: Tuxidow
2 Replies

7. UNIX for Advanced & Expert Users

Apache-Reverse proxy and load balancing

Hi All, I have a webpage loaded on server1 with authorization enabled by .htaccess, which can be accessed by http://ipofserver1/index.html. Now im planning a high availabilty load balancing in such a way that if the server1 is down due to some reason it should connect to another server. i have... (1 Reply)
Discussion started by: Tuxidow
1 Replies

8. UNIX for Advanced & Expert Users

Apache process cause huge load

Hello ! I have a problem with an apache process that is causing huge load. It starts from time to time - I'm not sure what is making it start beacause there's nothing in cron, but it appears every few minutes - and when it starts is uses a lot of RAM (up to 1.3GB) and create a huge load on... (1 Reply)
Discussion started by: Sergiu-IT
1 Replies

9. Shell Programming and Scripting

Need help in wrting Load Script for a Load-Resume type of load.

hi all need your help. I am wrting a script that will load data into the table. then on another load will append the data into the existing table. Regards Ankit (1 Reply)
Discussion started by: ankitgupta
1 Replies

10. HP-UX

Module jk2 failed to load into apache server!!

Hi There, Does anyone knows what could be the problem if my apache server is running but unable to load JK2 module into the server. My HTTPS is running on Apache Server 2.0.49 with ssl enabled and compiled with Mod_ssl on HPUX-11.11i. In fact we have try out the following parameter, ... (4 Replies)
Discussion started by: e_jeffhang
4 Replies
Login or Register to Ask a Question