It could be a filesystem problem: ssh does not like it when the filemode is set too generous. It should be "600", the important thing being "00" for group and world. Depending on your ssh version and the architecture you're running this might be a problem.

I hope this helps.

bakunin