Red Hat

Hi Experts,

I have been using Linux (Redhat server 5.6) as a NFS server and it is mounted to 4 linux (SUSE) servers for storing the db logs. I added an entry in fstab in all my 4 clients to make sure the mount is available all the time. but unfortunately the mount is not showing in servers after a day or 2. but I could see an entry in /etc/mtab saying that the mount is available but when I try df command it hungs.

I tried all most all the options like, -o nfsvers=3, tcp..... but still nothing works.

mount command output:

Code:

securityfs on /sys/kernel/security type securityfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
proc on /var/lib/ntp/proc type proc (rw)
t07lap00025.test.com:/dbnfs on /nas/dbnfs type nfs (rw,tcp,addr=xxx.xxx.xx.xxx)

mtab entry

Code:

securityfs /sys/kernel/security securityfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
proc /var/lib/ntp/proc proc rw 0 0
t07lap00025.test.com:/dbnfs /nas/dbnfs nfs rw,tcp,addr=xxx.xxx.xx.xxx 0 0

df command output

Code:

/dev/mapper/appsvg00-tibcolv02
                       5160576   1511988   3386444  31% /var/tibco

it hungs and not displaying /nas/dbnfs

when I mount manually using mount command it mounts but it is not displayed in df and hungs.
Any help to resolve this is highly appreciated.

Your NFS mount is working properly, it's just the df command which is getting stuck.

A work around would be to exclude the NFS mounts while using df:

Code:

df -l

If you want to see the disk space consumption of the NFS mount, try this instead of df:

Code:

du -sh /nas/dbnfs

Can you post the options you used to mount the NFS share?

Code:

grep dbnfs /etc/fstab

Also, how many NFS shares are mounted on this specific machine? It is possible that the df is getting stuck for another NFS share.

One way to get out of this kind of issue would be to use the "soft" mount option. But, if you do not want inconsistent data, use "hard,timeo=600" options. The timeo option specifies the timeout value (one tenth of a second) before it retries to get a respond from the server. Also, you may specify retrans option. By default, it is 3.

To further investigate, I would unmount the NFS share and mount it on a different client and check. This would tell us if the issue is from the client or the server.

Well, the NFS mounted in other servers and I can get into the NFS directory (read,write...) only I see issue with this server at this moment.

df hungs yes, but I cant roam into the NFS shared directory. cd /nas/dbnfs , ls /nas/dbnfs ..... everything hungs. ( I believe the mount is having issues only because of that the df,cd ,ls are getting hung)

I tried rw,soft option in fstab but nothing works...

current fstab option:

Code:

t07lap00023.test.com:/dbnfs /dbnfs nfs defaults 0 0

df, cd, ls -- all use the system calls to read a directory. This situation happens when the client doesn't get a response from the server.

Have you checked by manually mounting the share? does it throw any error? soft and timeo options are not remedy to the problem, they are just work around.

when I try to mount manually i`m not getting any error but still I cant read the files in the nas directory.

I could see the below logs,
Jun 3 09:59:59 t01lgpxxxxx kernel: nfs: server t07lap00025.test.com not responding, still trying

FWIW when you do an ls on an nfs mount that's gone stale, linux will attempt to remount it, it's at this point that it may be getting hung at so it might be wise to start a binary tcpdump between the two nodes, do an ls on the mount, let it hang for a bit, stop the tcpdump, load the packet capture into wireshark and see if anything jumps out at you. Basically I think we need more visibility on what exactly is going on between the hosts (is it making the three way handshake, is the other server not responding at all, are we getting a lot of dropped packets, etc).

I could see the below logs in NFS server and client, the NFS server log says that the mount request is authenticated. But as per client log it did not get reponse.

log in NFS server:

Code:

Jun 11 13:58:04 t07lap00023 mountd[26733]: authenticated mount request from xx.xx.xxx.xx:793 for /dbnfs (/dbnfs)

Log in NFS client:

Code:

Jun 11 13:58:39 t01lgp00032 kernel: nfs: server t07lap00023 not responding, timed out

--

Moderator's Comments:

Please view this link: How to use [code]...[/code] tags