Got a problem been going on for days now with 'mkdir' failing to create a new subdirectory on NFS shared fileystem located on our old primary Solaris NFS server. Getting message:

Failed to make directory "/dist/local/worklib"; Operation not applicable

In truss the error looks like:

13639: mkdir("/dist/local/worklib", 0777) Err#89 ENOSYS

Restarting automount apparently fixes this issue for many. Yet for our situation automount is not involved and does not fix.

For those stuck with mkdir failing and automount is not the problem, give statd restart a try. Note statd in question here is on the NFS server (not the clients).

I came up with this solution after no luck reading other posts almost exclusively tagging automount as issue (yet automount is not being used here and is not causing problem). Reboot is not desired. As it turns out the NFS client daemons on the NFS server in question were the cause. We got off the hook without a reboot or having to unmount and fsck. That's right, the daemon in question for this specific problem was statd. For an NFS server this was unexpected. Solaris 8 has /etc/init.d/nfs.client for restarting statd and other NFS client daemons... So:

sh -c '/etc/init.d/nfs.client stop; /etc/init.d/nfs.client start'

Above restarts statd nicely (among other NFS client processes) and immediately fixed mkdir.

Almost entire shop is Solaris 10 with this exception. NFS servers requires a lot of planning to restart... so Solaris 8 it remains for at least a little while longer.

For some additional background:

Most postings discuss restarting automount which does not fix this particular issue (sigh). My NFS server has been running 966 days and all 300+ servers in our environment use the affected filesystem heavily via NFS... if I reboot that will cascade into many failing applications. And I do not want to restart NFS as that will create an outage too.

Not being able to create subdirectories is a BIG problem! I cannot create the subdirectory on the NFS server nor any of the clients. And the filesystem is shared read-write. This fileystem is ufs and was recently grown via "/usr/lib/fs/ufs/mdkir -M". However after the filesystem was grown this issue was not affecting creating subdirectories so the correlation with expanding a filesystem and mkdir failing after several weeks is weak.

Searching internet for related posts so far have no solutions other than reboot and/or restarting automounter. FSCK is not an option as that requires an unmount to be safe -or- risk panicking the server.

I am 100% sure this was NFS client daemon, and 99% sure specifically restarting statd is all that is needed.