Well I can address some of your questions...
AIX - Advanced Interactive eXecutive
It IS UNIX. It was originally based on the AT&T UNIX SVR4
kernel. Yes it will run DR5 and yes it will run sendmail.
sendmail usually ships with AIX, Domino does not (costs extra).
sendmail on AIX runs absolutely fine however, your organization
may have some internal requirements that Domino could
fulfill and sendmail cannot. Domino can be more closely
compared to MS-Exchange in the fact that you get more of
a "work-group" environment (i.e. personal calendars, public
calendars, etc, etc). Domino fully supports SMTP as well.
Here's a good IBM link for more detailed info...
http://www-1.ibm.com/servers/eserver/iseries/domino/