War Stories

We have satellite modems and small servers installed in plenty of odd, rough, remote rural places, and satellite modems can be persnickety beasts. The provider claims they're all identical but some hand out static IP's over dhcp, some don't. Some respond to DHCP, some don't. They all have a nice parse-able web interface for runtime statistics, but not always the same parse-able web interface for runtime statistics. Trying to deal with their eccentricities in an automatic way can be a bit troubling if you want a uniform set of systems.

With a bit of wrangling I found an arrangement that worked in most situations. Usually the modems give a static address over DHCP, or a NAT address over DHCP, but a few odd ones don't respond at all, so I used dhcp failovers to auto-set the server's WAN port to an 192.168.x.x address when DHCP failed. Finally I could have the same settings work on all of my servers and modems, with actual static IP's when available, and it was good.

A little while later, a minor but important security fix came through for dhcpcd. In-house testing showed it properly picking up IP's, and the rest of the changes looked absolutely minor, so I deployed it. Most servers kept running... a few didn't. 100km later I was sitting on a bucket in a grimy, tick-filled wooden shack trying to figure out what the machine in front of me was doing to itself.

Code:

$ ifconfig wan

wan       Link encap:Ethernet  HWaddr 00:08:C7:E3:A1:13
          inet addr:169.254.32.239  Bcast: ...

The not-so-insignificant new feature turned out to be zeroconf support, enabled by default. This means failure is a perfectly acceptable outcome for dhcpcd now. On timeout it sets a useless random IP and returns success. No error, no failover; no failover, no network connection. One extra flag in dhcpcd's default options, a few long drives, and everything worked fine again.

I don't want to kill whoever decided zeroconf was a sane default for the world, but I'd at least like to pie him.