UNIX for Advanced & Expert Users

I recently ran into this problem. The solution offered here (and elswhere on the Internet) insufficiently failed to answer it. I had to dig deep to find it.

Depending on the precise server configuration, you might also see one of the following messages:

Code:

mount.nfs: access denied by server while mounting 10.0.2.2:/

Code:

mount.nfs: No such process

Code:

mount.nfs: Interrupted system call

For all of these error messages, the solution was the same.

It probably happens because of a bug in the NFS client or server shipped with 64-bit RHEL6 / CENTOS6 (nfs-utils 1.2.3-36.el6, nfs-utils-lib 1.1.5-6.el6), the one which supports NFSv4 and by default, requires the remote client to connect from a port number < 1024. The client NFS program refuses to use a low (<1024) port number, even if given the "resvport" option. The actual error you get depends on whether the server or the rpcbind daemon configuration (/etc/sysconfig/nfs on rhel6/centos6) has NFS 2 & 3 disabled, and which options are enabled for the IP shares exported.

Another quirk to my configuration is that I was trying to mount from a VM a share served by the hypervisor. So:

Code:

---------------
|  Host  H     |
|              |
|  ----------  |
| |  Host C1 | |
|  ----------  |
|  ----------  |
| |  Host C2 | |
|  ----------  |
---------------

Host C1 and C2 were trying to mount a share S served by host H.

The solution
Again, appropriate for version 4, as tested with RHEL6.4, is to specify "insecure" in the exports file for all IPs allowed to share that resource.

Code:

/export/shared   10.1.1.0(rw,insecure) 127.0.0.1(rw,insecure)

-------

Note: Many recipes/solutions (including the one offered above) call to add "fsid=0", but this is necessary and appropriate only if you want to mount the volume as that host's "root" filesystem. If that solved the problem (which it shouldn't have) then the problem lies elsewhere -- possibly mounting the wrong share-path or a path-permissions problem (a guess).

Note 2: The man pages indicate you can specify the mount program to use a reserved port, and I tried this, but my version refused to use a low port even using the resvport option. This might be due to a kernel configuration specific to RHEL6.

Note 3: You can use mount's -v option to see what happens when the mount fails. In the pathalogical case, the client tries V4, then V3, then V2 before giving up:

Code:

# mount -v 10.0.2.2:/ /mnt/share/
mount: no type was given - I'll assume nfs because of the colon
mount.nfs: timeout set for Tue Nov 12 18:15:35 2013
mount.nfs: trying text-based options 'vers=4,addr=10.0.2.2,clientaddr=10.0.2.15'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'addr=10.0.2.2'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.0.2.2 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: prog 100003, trying vers=2, prot=6
mount.nfs: trying 10.0.2.2 prog 100003 vers 2 prot TCP port 2049
mount.nfs: portmap query retrying: RPC: Program/version mismatch
mount.nfs: prog 100003, trying vers=2, prot=17
mount.nfs: trying 10.0.2.2 prog 100003 vers 2 prot UDP port 2049
mount.nfs: portmap query failed: RPC: Program/version mismatch
mount.nfs: prog 100003, trying vers=2, prot=6
mount.nfs: trying 10.0.2.2 prog 100003 vers 2 prot TCP port 2049
mount.nfs: portmap query retrying: RPC: Program/version mismatch
mount.nfs: prog 100003, trying vers=2, prot=17
mount.nfs: trying 10.0.2.2 prog 100003 vers 2 prot UDP port 2049
mount.nfs: portmap query failed: RPC: Program/version mismatch
mount.nfs: No such process

You can run a packet sniffer on the server to see that the remote side initiates a connection from a high port number:

Code:

18:15:07.350172 IP localhost.55088 > localhost.nfs: Flags [S], seq 2207422108, win 32792, options [mss 16396,sackOK,TS val 336392919 ecr 0,nop,wscale 7], length 0