NFS with a NAS: permanently inconsistent directory state across clients


 
Thread Tools Search this Thread
Operating Systems Solaris NFS with a NAS: permanently inconsistent directory state across clients
# 1  
Old 03-19-2015
NFS with a NAS: permanently inconsistent directory state across clients

Hi,

I am having some NFS directory consistency problems with the below setup on a local (192.) network:
1. Different permissions (chmod) for the same NFS dir are reflected on different clients.
2. (more serious) an NFS dir created on client1 cannot be accessed on client2; this applies to some directories, not others; when this problem applies to a directory, it is a consistent problem.



Setup:

NFS server: Thecus N8800, 16Tb raw, RAID6
Client1: Sun Fire V210, Solaris 5.10 Generic_139555-08
Client2: Sun Fire V100, Solaris 5.10 Generic_118822-23

Both clients nfs-mount. Flags: vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,acl,rsize=32768,wsize=32768,retrans=5,timeo=600
Attr cache: acregmin=3,acregmax=60,acdirmin=30,acdirmax=60



Use case on Client1:

CD to an nfs subdir:
Code:
cd /.../nfsdir
ls -la
drwxrwx---+ 56 user group   12288 Mar 13 15:28 .
drwxrwx---+  3 user group      30 Mar 17 13:57 ..
drwxrwx---+  3 user group   53248 Oct  3 04:41 somedir1
drwxrwxrwx+  7 user group    4096 Mar 13 15:29 somedir2

All good. CD to somedir1 works; can LS and see files. Same for somedir2. Note: somedir2 was mkdir'ed on Client1.



Use case On Client2:

CD to the same nfs subdir. Listing files works, but the permissions are different than what is listed on Client1:
Code:
cd /.../nfsdir
ls -la
drwx------+ 56 user group     12288 Mar 13 14:28 .
drwx------+  3 user group        30 Mar 17 12:57 ..
drwx------+  3 user group     53248 Oct  3 04:41 somedir1
drwx---rwx+  7 user group      4096 Mar 13 14:29 somedir2

PROBLEM1: the group permission for the same dir is different on client1 vs. client2.

CD to somedir1 works; can LS and see files.
PROBLEM2: *cannot* CD to somedir2:

Code:
bash: cd: somedir2/: Not a directory

On both clients, 'group' is defined in /etc/groups with the same id; 'user' is defined in /etc/passwd with the same id.

When I un-mount and re-mount the nfs dir on client2, I am able to access the directory in question (subdir2). Permissions, however, are still different across clients.



Does anyone have suggestions as to what is going wrong with my NFS setup? I'll be happy to post more information.

Thanks a lot!
# 2  
Old 03-19-2015
What is your NFS server? What does nfsstat show on the server and on each client?
# 3  
Old 03-19-2015
NFS server is a Thecus N8800 Network Attached Storage. It runs Linux and is configurable via a web interface. Unfortunately, I don't see much useful info via the web UI on NFS status. I also have not tried rooting the box.

"nfsstat -m" on the client is in my original post.

nfsstat on client:


Code:
Server rpc:
Connection oriented:
calls      badcalls   nullrecv   badlen     xdrcall    dupchecks  dupreqs    
48         0          0          0          0          0          0          
Connectionless:
calls      badcalls   nullrecv   badlen     xdrcall    dupchecks  dupreqs    
0          0          0          0          0          0          0          

Server nfs:
calls     badcalls  
0         0         
Version 2: (0 calls)
null     getattr  setattr  root     lookup   readlink read     wrcache  
0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     
write    create   remove   rename   link     symlink  mkdir    rmdir    
0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     
readdir  statfs   
0 0%     0 0%     
Version 3: (0 calls)
null        getattr     setattr     lookup      access      readlink    
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%        
read        write       create      mkdir       symlink     mknod       
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%        
remove      rmdir       rename      link        readdir     readdirplus 
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%        
fsstat      fsinfo      pathconf    commit      
0 0%        0 0%        0 0%        0 0%        
Version 4: (0 calls)
null                compound            
0 0%                0 0%                
Version 4: (0 operations)
reserved            access              close               commit              
0 0%                0 0%                0 0%                0 0%                
create              delegpurge          delegreturn         getattr             
0 0%                0 0%                0 0%                0 0%                
getfh               link                lock                lockt               
0 0%                0 0%                0 0%                0 0%                
locku               lookup              lookupp             nverify             
0 0%                0 0%                0 0%                0 0%                
open                openattr            open_confirm        open_downgrade      
0 0%                0 0%                0 0%                0 0%                
putfh               putpubfh            putrootfh           read                
0 0%                0 0%                0 0%                0 0%                
readdir             readlink            remove              rename              
0 0%                0 0%                0 0%                0 0%                
renew               restorefh           savefh              secinfo             
0 0%                0 0%                0 0%                0 0%                
setattr             setclientid         setclientid_confirm verify              
0 0%                0 0%                0 0%                0 0%                
write               release_lockowner   illegal             
0 0%                0 0%                0 0%                

Server nfs_acl:
Version 2: (0 calls)
null        getacl      setacl      getattr     access      getxattrdir 
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%        
Version 3: (0 calls)
null        getacl      setacl      getxattrdir 
0 0%        0 0%        0 0%        0 0%        

Client rpc:
Connection oriented:
calls      badcalls   badxids    timeouts   newcreds   badverfs   timers     
35344415   0          0          0          0          0          0          
cantconn   nomem      interrupts 
0          0          0          
Connectionless:
calls      badcalls   retrans    badxids    timeouts   newcreds   badverfs   
6          1          0          0          0          0          0          
timers     nomem      cantsend   
3          0          0          

Client nfs:
calls     badcalls  clgets    cltoomany 
35090121  1         35089942  77        
Version 2: (5 calls)
null     getattr  setattr  root     lookup   readlink read     wrcache  
0 0%     3 60%    0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     
write    create   remove   rename   link     symlink  mkdir    rmdir    
0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     0 0%     
readdir  statfs   
0 0%     2 40%    
Version 3: (35000040 calls)
null         getattr      setattr      lookup       access       readlink     
0 0%         6894383 19%  105714 0%    295215 0%    87658 0%     0 0%         
read         write        create       mkdir        symlink      mknod        
13428324 38% 13929856 39% 34582 0%     1829 0%      0 0%         0 0%         
remove       rmdir        rename       link         readdir      readdirplus  
2988 0%      147 0%       117 0%       0 0%         423 0%       4927 0%      
fsstat       fsinfo       pathconf     commit       
76 0%        4 0%         0 0%         213797 0%    
Version 4: (0 calls)
null                compound            
0 0%                0 0%                
Version 4: (0 operations)
reserved            access              close               commit              
0 0%                0 0%                0 0%                0 0%                
create              delegpurge          delegreturn         getattr             
0 0%                0 0%                0 0%                0 0%                
getfh               link                lock                lockt               
0 0%                0 0%                0 0%                0 0%                
locku               lookup              lookupp             nverify             
0 0%                0 0%                0 0%                0 0%                
open                openattr            open_confirm        open_downgrade      
0 0%                0 0%                0 0%                0 0%                
putfh               putpubfh            putrootfh           read                
0 0%                0 0%                0 0%                0 0%                
readdir             readlink            remove              rename              
0 0%                0 0%                0 0%                0 0%                
renew               restorefh           savefh              secinfo             
0 0%                0 0%                0 0%                0 0%                
setattr             setclientid         setclientid_confirm verify              
0 0%                0 0%                0 0%                0 0%                
write               
0 0%                

Client nfs_acl:
Version 2: (1 calls)
null        getacl      setacl      getattr     access      getxattrdir 
0 0%        0 0%        0 0%        1 100%      0 0%        0 0%        
Version 3: (90042 calls)
null        getacl      setacl      getxattrdir 
0 0%        90042 100%  0 0%        0 0%

# 4  
Old 03-19-2015
Umm, no you didn't provide nfsstat data in your original post:

Quote:
Originally Posted by cosmojetz
Hi,

I am having some NFS directory consistency problems with the below setup on a local (192.) network:
1. Different permissions (chmod) for the same NFS dir are reflected on different clients.
2. (more serious) an NFS dir created on client1 cannot be accessed on client2; this applies to some directories, not others; when this problem applies to a directory, it is a consistent problem.



Setup:

NFS server: Thecus N8800, 16Tb raw, RAID6
Client1: Sun Fire V210, Solaris 5.10 Generic_139555-08
Client2: Sun Fire V100, Solaris 5.10 Generic_118822-23

Both clients nfs-mount. Flags: vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,acl,rsize=32768,wsize=32768,retrans=5,timeo=600
Attr cache: acregmin=3,acregmax=60,acdirmin=30,acdirmax=60



Use case on Client1:

CD to an nfs subdir:
Code:
cd /.../nfsdir
ls -la
drwxrwx---+ 56 user group   12288 Mar 13 15:28 .
drwxrwx---+  3 user group      30 Mar 17 13:57 ..
drwxrwx---+  3 user group   53248 Oct  3 04:41 somedir1
drwxrwxrwx+  7 user group    4096 Mar 13 15:29 somedir2

All good. CD to somedir1 works; can LS and see files. Same for somedir2. Note: somedir2 was mkdir'ed on Client1.



Use case On Client2:

CD to the same nfs subdir. Listing files works, but the permissions are different than what is listed on Client1:
Code:
cd /.../nfsdir
ls -la
drwx------+ 56 user group     12288 Mar 13 14:28 .
drwx------+  3 user group        30 Mar 17 12:57 ..
drwx------+  3 user group     53248 Oct  3 04:41 somedir1
drwx---rwx+  7 user group      4096 Mar 13 14:29 somedir2

PROBLEM1: the group permission for the same dir is different on client1 vs. client2.

CD to somedir1 works; can LS and see files.
PROBLEM2: *cannot* CD to somedir2:

Code:
bash: cd: somedir2/: Not a directory

On both clients, 'group' is defined in /etc/groups with the same id; 'user' is defined in /etc/passwd with the same id.

When I un-mount and re-mount the nfs dir on client2, I am able to access the directory in question (subdir2). Permissions, however, are still different across clients.



Does anyone have suggestions as to what is going wrong with my NFS setup? I'll be happy to post more information.

Thanks a lot!
What's the inode number of the problem directories and/or files?

I suspect the problem is with the server - Solaris is VERY particular about NFS implementations being EXACTLY according to the NFS spec.

Can a 64-bit process access the files/directories?
# 5  
Old 03-19-2015
Ensure the mount points are directories with 755 permission! (umount to check.)
# 6  
Old 03-20-2015
Quote:
Originally Posted by MadeInGermany
Ensure the mount points are directories with 755 permission! (umount to check.)
That too.

I see that nfsstat data and I see non-zero values in NFSv2 stats. Is the server Linux-based? Or Irix-based - nothing like inode numbers greater than 2 gig - the files are invisible to 32-bit applications running on Solaris since an inode number above 2 gig violates the specs for a 32-bit system.
# 7  
Old 03-20-2015
The inode number of the problem file in question is: 1080094

Yes, the NFS server is Linux-based.

Re. ensure mount points are 755: yes, they are; otherwise I wouldn't be able to create any directories/files. Note that the problem occurs only with some newly created directories, but when it does occur, it is consistent (directory can never be read).

Regarding the other problem (perhaps related): even with freshly mounted nfs dirs, the two clients show different group permissions for all files/dirs:

Example of same dir:
Client1:
drwxrwx---+ 58 user group
Client2:
drwx------+ 58 user group

Both user and group are defined with same ids in /etc/passwd, /etc/group. What config might be faulty to warrant this behavior?

Thanks.
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Solaris 9 Home Directory, Two Machines Sharing a NAS

Good Morning, I have 2 Solaris 9 machines sharing a NAS, and need to have users to be able to log in from the 2nd machine and get to all of their files on the NAS that were created on the 1st machine. So far its working ok, but when users log in to the second machine, their user IDs show... (20 Replies)
Discussion started by: Stellaman1977
20 Replies

2. Red Hat

Add directory to path permanently

I ssh in and am trying to add a directory permanently to $PATH in centos 7 and having issues. My current $PATH is echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin but when I do a sudo nano ~/.bashrc # .bashrc # User specific aliases and functions alias... (7 Replies)
Discussion started by: cmccabe
7 Replies

3. UNIX for Dummies Questions & Answers

NFS / DAS / SAN / NAS - Which is best?

i've used only NFS and as many already found out, it can be or rather i should say, it is very unreliable. based on the collective experiences of the members on this board, i would really appreciate it if someone can tell me what the next best file sharing method is? is it DAS? SAN? NAS? ... (1 Reply)
Discussion started by: SkySmart
1 Replies

4. UNIX for Dummies Questions & Answers

Unmounting NFS idle clients from server

Hello World, We have a software repository server in our environment which we use as an NFS server. Now this has been going on well before I was hired. Now, I observed many users not unmounting the NFS resources after their use. I ran showmount and it showed 513 current sessions. :wall: Is... (7 Replies)
Discussion started by: satish51392111
7 Replies

5. AIX

Open firmware state to running state

Hi Admins, I am having a whole system lpar in open firmware state on HMC. How can I bring it to running state ? Let me know. Thanks. (2 Replies)
Discussion started by: snchaudhari2
2 Replies

6. UNIX for Advanced & Expert Users

Quest on NFS with 1 Server & 2 Clients

Hello; I work now with a team which has based her "applications" on the following schema: 1 NFS client C1 (linux redhat 5) writes a file fic.dat on a NFS SERVER S (RH 5) Another NFS client C2 is waiting for the same file fic.dat (on NFS server S1), and, when "fic.dat" appears, then makes... (8 Replies)
Discussion started by: SolarMax
8 Replies

7. UNIX for Dummies Questions & Answers

NFS mount of Windows NAS

The UPS connected to the Disk Array portion of my Windows 2003 NAS burned up over the weekend. Reconnected it to a new UPS and re-booted the NAS box. Since then I have not been able to get my HPUX 10.2 box to mount the shared drives on the NAS. At boot, the NFS client & server subsystems do a... (0 Replies)
Discussion started by: twalker0
0 Replies

8. UNIX for Advanced & Expert Users

NFS to NAS transfer

Hey guys, My servers run Solaris 10. I have to move data from my NFS mount drives into the NAS storage area with all the same permissions ( for the users and programs and everything else). Can any one help me please ?? Thanks. MisterKhan (0 Replies)
Discussion started by: MisterKhan
0 Replies
Login or Register to Ask a Question