From Sunsolve:
Question: Does Solaris have a feature that will lock a user account after 3
or more failed login attempts?
Document Body: Top
Solaris[TM] 9 and older versions do not have a "built in" feature that allows you to lock or disable an account after a number of failed logins. However, they do have the capability to accept a pluggable authentication module. Granting the flexibility for such capabilities to be customized into older versions of Solaris via PAM.
One reason why this was not initially included was because it opens the possibility for "denial of service" attacks for users like root, staff and other. It is supported and included in Trusted Solaris because the root user is not a regular UNIX user, rather it is a role and cannot be logged into directly.
Systems using LDAP as their naming service are able to achieve this functionality in conjunction with the latest LDAP client patches and Sun[TM] ONE Directory Server 5.1 or newer.
Starting with Solaris[TM] 10, the option to configure this is available. This is done using the /etc/user_attr database and/or /etc/security/policy.conf to set lock_after_retries. The account will be locked after the number of retries is met as defined by RETRIES, located in the /etc/default/login file.
If there is still an interest in setting this up for older versions of Solaris, it is considered customization. You can either pay to have the customization, write it yourself, or search the internet for free PAM modules. All three are not supported by Sun Support. However, you can contact Sun Professional Services for information on what kind of service and fee they provide for this.
Here is Sun's link for PAM information:
http://wwws.sun.com/software/solaris/pam/
Reference:
RFE 4524783 enhance PAM authentication to allow account locking.
This feature is integrated in Solaris 10.