Hello,
I administer a bunch of Apple XServes running OS X Server and I have one in particular that is annoying me since I brought it online.

The host is a dual quad core Intel CPU. en0 is attached to a routable network and en1 is attached to a non-routable network (private switch that all the hosts in our rack use to communicate with each other for administrative tasks). It also has a PCI 2 port ethernet card installed as well. The networking is manually defined and there is no restriction on the network for mapping MACs to IPs (e.g. any host can use any IP in our range through any active port in the switch). We have IPv6 disabled for all the interfaces.

I noticed the other day that I seemed to be having network latency issues with en0 (not uncommon on our network, but this was happening more than usual). I started running a ping against the host and noticed that it would run sub .5ms as expected then jump to .5s for a few pings (<20) and then go back down. In some causes i've caught the ping time as high as 1.7s.

These erratic ping times occur even when pinging from a host on the same switch in the same rack, but do not occur when pinging en1.

I believe this is a software issue as I have:
1) Swapped cables and switch ports with another host that does not exhibit any issues. Both hosts continued to perform as they did before the swap.

2) Moved the en0 network to one of the PCI ports (en2) with no difference in the ping issues.

Because of the above tests, I believe the networking external to the host isn't at fault (e.g. tried different port on the host, different port on the switch, different cable, and no other host on that switch has issues).

I've checked the config for en0 and it appears to be configured correctly and inline with our other hosts.

I've watched MySQL, Apache, and general CPU load and none appear to correlate with the latency issues (e.g. high or no load seems to have an equal chance to exhibit the problem).

What else can I look at that can add insight into what is causing this and how to fix it?

Thanks,
-dave