UNIX for Advanced & Expert Users

All the servers are mission critical.
And in terms of risk management, we have an SLA of maximum 4 hours to bring the server back up in an event of a catastrophe.

Does not sound very "mission critical" to me.

If you define everything that can be down with an SLA for four hours as "mission critical", what would you define a server that if it went down it would cost the company 100K to 1M USD per hour?

Most people would not define a service as "MISSION CRITICAL" if it has a SLA of four hours, to be frank. But then again that depends on the "MISSION".

If you have SLA of four hours, then you can easily make a mistake and recover from it long before the four hour SLA window is reached. That is more like "A STANDARD BUSINESS SLA", for a lack of a better term.

Do you have a risk management team (normally a part of either the IT security or audit teams) responsible for the risk management of all these servers?

If so, get them involved.

The biggest loses any company has is usually a mistake by a well intended trusted employee. Often, these big mistakes are caused by trying to automate an upgrade to hundreds of devices (routers, servers, firewalls, etc).

Best to set up a test bed, work on the changes, and get it working. You cannot just take "YUM" and try to upgrade if the original installs were done manually. This is a formula for a lot of downtime!

Thank you for the clear and thorough information/explananation.
Yes, we do have IT Risk Management team. But I have no experience so far in getting them involved in any of the projects I have worked on.
What can they contribute if I get them involved?

Normally if we are making upgrades to mission critical applications in 100s of servers in a large organization with a risk management team we should notify them if if we plan any upgrades which may cause an outage.

You should test in a test bed, make a plan, identify the risks and notify your risk management team.

Upgrading the core "mission critical" application on 200-300 web servers requires planning and team work.

What happens if you make a mistake and bring down the application(s)? Do you want your organization to be caught off guard when customers are calling in, angry their service is down?

You have described a major infrastructure upgrade. You certainly do not want to upgrade with YUM over a manually installed configuration until you have completed tested this idea in a test bed, insure you have backups of each application and database before hand, etc. In other words, you need a plan in coordination with your IT risk management team, customer service, etc.

This is how we work as IT professionals.