Since nobody else has answered I'll take a shot at it for you (my scripting isn't always the best but this worked on my workstation).
I didn't know of any other way to do this - but you can't easily use /etc/security/failedlogin file because it isn't a regular text file (you have to use 'who' to read it).
1. add the following line to the end /etc/syslog.conf
auth.debug /logs/userauth.log rotate size 10m files 4
2. touch /logs/userauth.log
3. refresh -s syslogd
4. Create a script that constantly watches /logs/userauth.log for new lines - something like:
# vi logwatch.sh
LOG=/logs/userauth.log
echo "\n\n" >> ${LOG}
tail -1 -f ${LOG} |
while read LINE
do
case "${LINE}" in
failed)
echo ${LINE} | mailx -s "Failed login"
me@mail.com ;;
esac
done
5. Run your script and wait for the mail messages to start. You could put this in /etc/inittab to be started each time the system is rebooted.
HTH