Login or Register to Ask a Question and Join Our Community


Solaris

VCS Log

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Solaris VCS Log
# 1  
Old 04-09-2009
VCS Log
Hi All,

Can anyone help me to analyse this VCS log. One of my applications failover suddenly and I need to find out the reason behind this.

2009/04/09 10:58:57 VCS ERROR V-16-2-13067 (CS49PAPS2) Agent is calling clean for resource(Web-ebill2app) because the resource became OFFLINE unexpectedly, on its own.
2009/04/09 10:58:58 VCS NOTICE V-16-55008-10424 (CS49PAPS2) WebLogic9:Web-ebill2app:clean:VCSagentFW:SetupLogging:[clean] Entered by resource instance [Web-ebill2app] with clean reason [4][Unexpected Offline]
2009/04/09 10:58:58 VCS NOTICE V-16-55008-20080 (CS49PAPS2) WebLogic9:Web-ebill2app:clean:<WebLogic9::Kill>:WebLogic Server process not found.
2009/04/09 10:58:58 VCS INFO V-16-2-13068 (CS49PAPS2) Resource(Web-ebill2app) - clean completed successfully.
2009/04/09 10:59:00 VCS INFO V-16-1-10307 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (Not initiated by VCS)
2009/04/09 10:59:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:00 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for resfault; script doesn't exist
2009/04/09 10:59:00 VCS INFO V-16-1-10305 Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:01 VCS INFO V-16-1-10305 Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:02 VCS INFO V-16-1-10305 Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10305 Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:12 VCS ERROR V-16-1-10205 Group Ebill2ap-app is faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10446 Group Ebill2ap-app is offline on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating CS49PAPS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS INFO V-16-1-50010 Group Ebill2ap-app is online or faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating TS49SDAS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:12 VCS INFO V-16-1-10298 Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:13 VCS INFO V-16-6-15002 (CS49PAPS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline CS49PAPS2 Ebill2ap-app successfully
2009/04/09 10:59:13 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for lvmvg_postoffline; script doesn't exist
2009/04/09 10:59:13 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2009/04/09 10:59:13 VCS WARNING V-16-10001-1014 (TS49SDAS2) DiskGroup: Dg-ebill2app: online: Diskgroups will be imported without reservations
2009/04/09 10:59:15 VCS NOTICE V-16-10001-1009 (TS49SDAS2) DiskGroup: Dg-ebill2app: online:vxdg import succeeded on Disk Group ebill2apdg
2009/04/09 10:59:15 VCS NOTICE V-16-10001-1010 (TS49SDAS2) DiskGroup: Dg-ebill2app: online:Volumes in Disk Group ebill2apdg are started. Any mirrors are updated in background
2009/04/09 10:59:17 VCS INFO V-16-1-10298 Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:17 VCS NOTICE V-16-10001-11501 (TS49SDAS2) Volume:Vol-ebill2app: online:Volume ebill2apvol is started. Any mirrors are updated in background
2009/04/09 10:59:18 VCS INFO V-16-1-10298 Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:19 VCS INFO V-16-1-10298 Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:20 VCS WARNING V-16-55008-20158 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>:Server Process is not running.
2009/04/09 10:59:26 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:31 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:36 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:41 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:46 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:51 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:56 VCS NOTICE V-16-55008-20162 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: First Level Check , server responding at [192.168.100.41:7002].
2009/04/09 10:59:56 VCS NOTICE V-16-55008-20255 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<main::OnlineEntryPoint>:Resource [Web-ebill2app] started up successfully.
2009/04/09 10:59:58 VCS INFO V-16-1-10298 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:58 VCS NOTICE V-16-1-10447 Group Ebill2ap-app is online on system TS49SDAS2
2009/04/09 10:59:58 VCS NOTICE V-16-1-10448 Group Ebill2ap-app failed over to system TS49SDAS2
2009/04/09 10:59:59 VCS INFO V-16-6-15051 (TS49SDAS2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group
2009/04/09 10:59:59 VCS INFO V-16-6-15002 (TS49SDAS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart Ebill2ap-app successfully
2009/04/09 10:59:59 VCS INFO V-16-6-15004 (TS49SDAS2) hatrigger:Failed to send trigger for postonline; script doesn't exist
2009/04/09 15:46:39 VCS INFO V-16-1-50135 User root fired command: hagrp -clear Ebill2ap-app CS49PAPS2 from localhost
2009/04/09 15:46:39 VCS INFO V-16-1-10307 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (Not initiated by VCS)


thanks
# 2  
Old 04-09-2009
the VCS log only shows that the resource grp became offline in one of the host , thus initiating a failover. But it wont explain WHY it went offline.
# 3  
Old 04-09-2009
Java
Hi ronny_nch

I am no expert in this field, but have a look at the highlighted areas:

<snip>

2009/04/09 10:59:12 VCS ERROR V-16-1-10205 Group Ebill2ap-app is faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10446 Group Ebill2ap-app is offline on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating CS49PAPS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS INFO V-16-1-50010 Group Ebill2ap-app is online or faulted on system CS49PAPS2


2009/04/09 10:59:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:20 VCS WARNING V-16-55008-20158 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>:Server Process is not running.2009/04/09 10:59:26 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level



2009/04/09 10:59:59 VCS INFO V-16-6-15051 (TS49SDAS2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group
2009/04/09 10:59:59 VCS INFO V-16-6-15002 (TS49SDAS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart Ebill2ap-app successfully
2009/04/09 10:59:59 VCS INFO V-16-6-15004 (TS49SDAS2) hatrigger:Failed to send trigger for postonline; script doesn't exist

CS49PAPS2 (Not initiated by VCS)

TS49SDAS2 (VCS initiated)

<snip end>

I hope this is of help.

Jnike
# 4  
Old 04-13-2009
Hi Guys,

Thanks for the reply. Could weblogic be the reason because the server hosts few other applications as well but they did not failover.

rgds,
# 5  
Old 04-13-2009
Possible.. The cluster wont failover unnecessarily unless it hits issue with the applications like weblogic or oracle
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

VCS not active

Dear All, My current system run Solaris 9 SPARC 64 bit , and VXVM , VCS 4.1. I check all the node information . ls -l /etc/vx/*.exclude /etc/vx/*.exclude: No such file or directory root@devuardbs01 # vxdctl license All features are available: Mirroring Root Mirroring Concatenation... (0 Replies)
Discussion started by: linux_user
0 Replies

2. Solaris

VCS - 3 node - IP change

Hey Admins, I need steps to do an IP changes on 3-node VCS cluster. All IP's (public, cluster IP's, backup IP, etc.)want to change. We have a downtime scheduled. Please let me know the steps to do so. Thanks in advance. (5 Replies)
Discussion started by: snchaudhari2
5 Replies

3. Solaris

VCS- question

Hi admins, I have a question on shared diskgroup in vcs. Let's say i have two node cluster node A and B. I am having the failover service group (including the required resources such as disk grp, IP, mount..etc) For an instance, the service grp is running on node A and suddenly if one of the... (0 Replies)
Discussion started by: snchaudhari2
0 Replies

4. Solaris

VCS on Solaris: VCS ERROR V-16-2-13077 (host2) Agent is unable to offline resource(DiskReservation)

hi, dear all I get a problem "VCS ERROR V-16-2-13077 " on VCS 4.1 for Solaris 10. I can not offline the host2 when the raid is bad. I don't know the reason and how to offline host2 and switch to host1. please help me, thank you! the message of engine_A.log is : ... (2 Replies)
Discussion started by: ForgetChen
2 Replies

5. Solaris

VCS Clusters

:)Hi, can someone please explain VCS clustering and where do we need VCS clusters ..? :o:)Thanks in advance :o:) (1 Reply)
Discussion started by: amitbisht9
1 Replies

6. Solaris

VCS for Solaris 32 bit OS

Hi Admins, I know that VCS supports Solaris x86-64bit only...and I need vcs for Solaris 32 bit-x86..running on vmware workstation.. I need to run vcs on vmware workstation...And my laptop support only 32bit... Please suggest.. Thanks.. (1 Reply)
Discussion started by: snchaudhari2
1 Replies

7. UNIX for Dummies Questions & Answers

vcs agent

Hi all, I'm new to vcs. I have a doubt. I need to know, what will happen if an agent is stopped while reources being online. Eg.. while the oracle agent is stopped, will all the oracle resources will become offline.. Advanced thanks (1 Reply)
Discussion started by: sunshine12
1 Replies

8. Solaris

VCS heartbeat

we have a vcs cluster set up and noticed that they were showing one of the heart beat link qfe3 as DOWN. Both qfe2 and qfe3 were fine all along, btw if I were to push in/re-set the hb cable, do you know whether it will panic or cause anything to the system,? Its a DB critical prod nodes (3 Replies)
Discussion started by: incredible
3 Replies

9. High Performance Computing

VCS on AIX

Hi, is anybody here experienced in running VCS on AIX and happy to give me some advice regarding configuration questions? I expect very high network IO on my cluster - so a single gig-e network will not be sufficient - and though I would like to configure more than one network card per node and... (0 Replies)
Discussion started by: zxmaus
0 Replies

10. High Performance Computing

What does the 'probe' do in VCS?

Weird question I know, but I'm intrigued. Say for instance you have an Application in an SG with the usual start/stop/monitor. To successfully probe does it just check for existence of the script/file ? Thanks... (1 Reply)
Discussion started by: itsupplies
1 Replies
Login or Register to Ask a Question