UNIX for Advanced & Expert Users

I'm becoming a dying breed where I work. More and more sys admins are advocating automatically restarting failed services such as tomcat, jboss, etc. I've always been against doing this except with buggy apps that can't be fixed or avoided.

My main argument is that I feel it's a trick used by lazy sys admins who don't want to troubleshoot their apps. Almost everything we have that is customer facing is behind a load balancer (we have a lot of customers). If the LB is properly configured, it will pull a node out of the rotation if it fails a health check. If the pool is sized properly it will have at least n+1 servers running and should be able to handle the load if one node dies or is removed. I feel we should let the app fail, alert on it, remove it from the pool, and troubleshoot it to find out why. Turn up a new node to take it's place if necessary. If the bad app is auto restarted and it is indeed bad, we will continue to route customers to it and it could negatively affect them.

They argue that "apps just fail" and that we should restart them asap to keep them up and servicing customers.

I'm starting to feel like the old geezer of the group and these damn kids won't get off my lawn. If you wouldn't mind, please let me know your take on this. I'm not looking for everyone to agree with me and I'm not against changing my views. They just haven't provided a good argument.

Thanks,

MG

Let me put it this way: What function does leaving them down perform for you? Does restarting them prevent you from debugging them?

sysadmins push for maximal uptime. This is what they are paid for:

system availability
data security

Well, like I said, everything is in a pool, behind a load balancer. An app that fails is a broken app. Maybe it was just a glitch and a restart would remedy it, but "maybe" it's not. Maybe the app on that host is actually broke. If you restart it, the load balancer will continue to send traffic to it and those customers will be affected. To me that's a huge negative.

jim mcnamera said "sysadmins push for maximal uptime. This is what they are paid for". Where I work it's all about SLA's. If I'm sending customers to a malfunctioning node (or worse, sending the SLA monitor to a malfunctioning node) we take a hit on SLA. Big no-no here. So, yes, uptime is what we're paid for, but uptime for the service, not for the individual services running behind it. Plus, what if you had an app that crashes once a day but you autorestart it, resulting in 99.99% uptime. Would you seriously consider that a success?

MG

I don't know why you quoted me, you didn't answer either question.

The hard bit can be detecting that the application has failed. Just relying on the output from a single "ps" command is not safe because a busy system may give a blank or incomplete response to a "ps" command.

To paraphrase Corona688 there is no harm in installing a workaround while you find and repair the root cause ... or determine that the root cause cannot be repaired.

If for example you have a client-server application running on an unreliable network (like the Internet) there is a good case to configure a client retry mechanism backed with a carefully-designed automatic client restart combined with a matching dead-session cleanup in the server.

Uptime is gold.

I am a strong advocate of always having a watchdog process in place to watch all critical services and restart them if they go down.

Debugging why processes fail is another topic and certainly should not be used as an excuse to shave uptime down.