UNIX for Advanced & Expert Users

Hi, all.

I'm a unix sysadmin at a community college and I've been doing this for about 10 years. I've been thinking and I realised I spend most of my time worrying about upgrades. By upgrades, I mean anything where I have to interrupt service to switch to a different way of providing that service. So, I mean OS upgrades, software upgrades, hardware upgrades, moving stuff, configuration changes etc... etc.... I don't want downtime and I don't want to roll-out untested stuff or stuff that will surprise and confuse my users. So, I don't want to have downtime while I do an upgrade and then test it and I also don't want to upgrade stuff in place on a live server.

My general strategy has been to have two servers for everything. One serves content while the other gets upgrades and then I switch. Over the years many programs, systems, shortcuts, hacks and cludges have accumulated to make this better. And now, ten years later, I'm wondering if all those hacks and cludges are me re-inventing the wheel and there is a better overall strategy to this kind of thing, an upgrade management system or philosophy or discipline or something like that.

Like I said, I'm at a community college so we're pretty small, but I'm interested in what anyone of any size does.

Thoughts? What do you do?

Thanks!
-Pileofrogs

Well, if your apps can float on many hosts redundantly, then you can pull each host in rotation for upgrades with relative ease. Applications with persistent tcp connections and with state in server memory are a problem. Sometimes, you can put a web front end on them, or push the state into a flat file mmap'd to the apps running on many machines. Files should be SAN or mirrored between machines, so they do not disappear as you rotate machines in and out. An optimal configuration is for the hosts to be a sort of liquid resource, where you can add a host in and the storage, query, churn and CPU use is attracted to the new host until a balance is achieved.