UNIX for Advanced & Expert Users

Think about it.... what else could possibly happen if all 37 servers transmit UDP packets that attempt to arrive at the same time? But you could put protocol sniffers on all 37 servers and look at all syslogd packets that they send. And sniff the destination server and look at all arriving *intact* syslogd packets and compare the lists.

Bear in mind though that the packets may not even make it to the network. If you throw UDP packets at an interface faster than it can transmit them it will queue some up, then start discarding them. The same thing can happen on the other side. If the syslogd does not read incoming udp packets fast enough, they get tossed. I'm not sure but I think the the udp error counts get incremented by either condition. "netstat -s -P udp" will show those counts.

It might be interesting to us a sniffer on your loghost (snoop, tcpdump, snort), to watch udp traffic. Should be able to spot any malformed or incomplet packets. As well as the previously mentioned netstat command.

If that's not the case, maybe you could write a seperate syslogd app in Perl. I know nothing about you setup, but some of the messages could be sent to a different port on your loghost (on which the new Perl syslogd is listening).

http://search.cpan.org/~sparsons/Net....8.0/Syslog.pm

Also, maybe you could benchmark your loghost. Shutdown in network interfaces (so you aren't getting any network traffic) from the syslog clients. Then run a simple script which bombards syslogd which local messages. Count the number sent. Compare to the actual number logged.

#!/bin/sh

start=`date "+%M:%S"`
x=0

trap 'echo "";
echo "started at $start";
echo "finished at `date "+%M:%S"`";
echo "sent $x messages";
echo "found `grep TEST_MESSAGE /var/log/messages | wc -l` messages";
exit' 2

while()
do
logger -p syslog.notice TEST_MESSAGE_$x
x=`expr $x + 1`
echo sent $x
done

Running this script, my FreeBSD machine logged about 3000 messages in 1 minute. None were lost. If you run the script more than once, you will have to change the test log name for the count to be accurate.

If you truely want to beat your machine up, we can run a forking Perl script. I'll have to get back to you on that.