You can do a man on core -
NAME
core - core image file
DESCRIPTION
The operating system writes out a core image of a process
when it is terminated due to the receipt of some signals.
The core image is called core and is written in the
process's working directory (provided it can be; normal
access controls apply). A process with an effective user ID
different from the real user ID will not produce a core
image. This is also true for a process with an effective
group ID different from the real group ID. Set-user-ID and
set-group-ID programs do not produce core images either when they terminate, since this would cause a security loophole.
Also go to
www.sendmail.org and read the FAQs. If this server is for mail only, you would want to insure you turn off anything else (telnet, ftp, remote services...). Check the web for security issues.
Another thing - check the size of your /var/adm/messages* file(s).
It may be they are filling up your / file system. You wrote that /var/spool/mqueue was empty so it isn't email doing it.
Which email are you using - Sendmail or something else?
