Can HPOV monitor server hung state ?


 
Thread Tools Search this Thread
Special Forums UNIX and Linux Applications Infrastructure Monitoring Can HPOV monitor server hung state ?
# 1  
Old 06-26-2019
Can HPOV monitor server hung state ?

Hi,
We have Solaris-10 running on VMware (x86). It is being monitored by HP Openview. Sometimes when this server hungs, while ping still works, HPOpenview can't alert that server is down (which is actually unresponsive).
First symptom we see is, login failure. It will ask user name and after providing that, either it won't ask for password or if asks, it will just wait there and doesn't come on prompt.
I am not expert of mionitoring tool, so seeking suggestions here. Is there any capability in HPOpenview, which can track and alert this kind of hung state, like login failure ?
Thanks
# 2  
Old 06-26-2019
My guess is that this server is not in a hung state but is running dead slow. The fact that ping still works proves that the vmware guest OS is alive.

This guest OS is either out of memory resource or the whole platform is out of memory resource. Since you are not complaining that other guests are showing any signs of problems I would ask whether a maximum real memory limit was put on this guest when is was configured. If so, as load increases on this guest it could reach a level where it is allowed no more memory and so starts to page like mad. The symptoms of very slow (almost impossible) login are typical of resource limitation. If you were to wait a very long time probably login would eventually complete.

What OS is under VMWare and booting the platform?

Could another guest OS on this platform suddenly be demanding a lot of resource thereby killing other guests?

Could you easily increase the overall RAM on the whole platform to test this theory out?
# 3  
Old 06-26-2019
Yes, we can assume that it was dead slow. I waited for 20 minutes, before I hit reset button. since this was critical application server, I was not able to wait longer. From VMWare console, I checked and memory graph was not showing peak utilization. But I have seen behavior where server was frozen due to memory crunch and VMWare console doesn't show that. So your theory can be true in this case too.

VMWare platform is stable (over 300 VM servers are running on it) and none other VM complained about it and still have large amount of memory. Memory is capped to each VM. For example, for this affected VM, 8 GB of memory is allocated.

As a preventive action, I am looking for alert, if it happens again. There could be a small script running from other server and keep login to affected server and say "Login OK". As soon as it delays or not responding for 10-20 seconds, it will send email to admins. But we want to do it many servers and that will become a messy solution. HPOpenview is handled by different team, but they are not taking any initiative to advance on it. So I am looking for solution, if something can be done with this tool.
# 4  
Old 06-26-2019
HPOV like most centralized network management systems (NMS) rely primarily on polling to update their management system.

This means you cannot get information by polling an unresponsive server.

However, most of these same systems also can be configured to send traps.

This means, your HPOV team needs to set up your system to send traps back to the management system before the system being monitored slows to a crawl (soft failure) and cannot respond to polling,

In other words, most novices set up network management systems like HPOV to use only polling; but experienced network management people will also set up traps to be sent back to the NMS for certain critical processes which need to alert the NMS prior to overall system "failure" (also meaning a soft failure, not only a hard failure).

I have extensive hands-on experience in NMS including debugging HPOV when it was a black and white versus decades ago. All well configured NMS will be configured to poll (95% +) and to trap (< 5%) alerts.

Bottom Line: Set up traps for the critical processes to alert the NMS before the system soft-fails (slows to a crawl).

Hope this helps.

Cheers.
# 5  
Old 06-26-2019
This is good information to pass on to them.
But in this situation when system is dead slow on login (or completely dead on login), traps can be set on what processes ? I am tying to understand concept of setting up traps. Does a process login to sever (with some default credential) and come back with response ?
If we assume that there is soft failure on NIS service (ypwhich/ypbind), that is not happening here. As soon server goes into hung state, login doesn't go after providing password (or sometimes doesn't ask for password after giving username).
# 6  
Old 06-26-2019
You have to monitor the system and send out the traps on the critical processes you want to monitor before the system soft-fails.

It's that simple.
# 7  
Old 06-26-2019
I got your point. But probably it is difficult to know the process, which we assume, will start failing, before server start choking up. I will browse more in messages file, if that indicate some failure message, right before server hangs
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Flow control state changed in server logs

Hi, We observe below logs from switch - the database servers rebooted becaause they couldn't do I/O on vfiler -Any pointers looking at below logs please? Switch logs: 2016 Apr 30 07:41:16.729 EAG-ECOM-POD111GPU-SWF1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet152/1/8 is down (Link... (0 Replies)
Discussion started by: admin_db
0 Replies

2. Linux

Server hung, is this a stack trace?

Hi everyone, Our Red Hat server hung yesterday, and I managed to log into the console and see the following message: RIP: 0010: mwait_idle_with_hints+0x66/ 0x67 RSP: 0018:ffffffff80457f40 EFLAGS: 00000046 RAX: 0000000000000010 RBX: ffff810c20075910 RCX: 0000000000000001 RDX:... (6 Replies)
Discussion started by: badoshi
6 Replies

3. Red Hat

How to find the process which is caused system hung state?

when system is hung state due to swap, we will reboot it through ILO. i want to know which process caused system hung. (1 Reply)
Discussion started by: Naveen.6025
1 Replies

4. SuSE

Server hung with firmware error

Hi all We've had an issue over the weekend when one of the SUSE Linux Enterprise Server 11 hung and had to be rebooted. The thing is that I got the ticket alert for a FS exceeding its usage at about 22:41:49 PM on 23 March. I checked the dmesg, the messages log and the boot.msg but all I found... (1 Reply)
Discussion started by: hedkandi
1 Replies

5. Shell Programming and Scripting

How can i get the state of weblogic server

Hi, all Now i want write a shell to get the state of weblogic server,and when the Managed Server's state is not ok, after 3 times checking, i will send msg to the system administrator by sms. BTW, my environment is : Linux ,Redhat 5.4 64bit weblogic version: 10.3.3 the count number... (1 Reply)
Discussion started by: wangsk
1 Replies

6. Solaris

HPOV in solaris

Good Day all i have a solaris 8 server and i want the procedure for how to install HPOV becuse dont have any small info about solaris . (1 Reply)
Discussion started by: thecobra151
1 Replies

7. Solaris

How to clear maintenance state for apache2 server?

Can any one of you suggest me the method to get apache server in online from maintenance mode. I tried in the following way, but couldn't get that service to online. bash-3.00# svcs -a | grep apache legacy_run 9:51:55 lrc:/etc/rc3_d/S50apache offline 9:51:22... (3 Replies)
Discussion started by: Sesha
3 Replies

8. HP-UX

ssh session getting hung (smilar to hpux telnet session is getting hung after about 15 minutes)

Our network administrators implemented some sort of check to kill idle sessions and now burden is on us to run some sort of keep alive. Client based keep alive doesn't do a very good job. I have same issue with ssh. Does solution 2 provided above apply for ssh sessions also? (1 Reply)
Discussion started by: yoda9691
1 Replies

9. UNIX for Advanced & Expert Users

close_wait connections causing a server to hung

Hi Guys, Just wondering if anyone of you have been in a situation where you end up having around 100 close_wait connections and seems to me those connections are locking up resources/processes in the server so unless the server is rebooted those processes won't be released by the close_wait... (3 Replies)
Discussion started by: hariza
3 Replies

10. HP-UX

Server hung

So my server was hung when I came in this morning. It was responding to pings, but the console and telnet sessions would not respond. There was no disk activity. The display said FA1F which I discovered that the "A" represents a high CPU load. I tired several things to get it going but was forced... (6 Replies)
Discussion started by: biznatch
6 Replies
Login or Register to Ask a Question