Authentication related settings are stored in /etc/sysconfig/authconfig. The default is MD5.

Look at Article 16972 in the Redhat Knowledgebase for details of how to figure out which algorithm was used.