GPFS file system corrupt issue


 
Thread Tools Search this Thread
Operating Systems AIX GPFS file system corrupt issue
# 1  
Old 08-17-2010
GPFS file system corrupt issue

Hi Administrators,

I have one issue related to GPFS filesystem. We have bad entries in this file system, which reflects error like
Code:
ls: 0653-341 The file <filename> does not exist.

when we give "ls -ltr" on this directory.

So we taken the FS filesystem offline and followed the below steps.

Code:
root@profrd06:/root]# umount /opt/minotaur/Data/Shared_CFDX
root@profrd06:/root]# mmfsck /dev/gpfslv
Checking "gpfslv"
Checking inodes
Checking inode map file
Checking directories and files

Error in directory inode 461241:  DirEntryBad
Directory entry "GSM_20080821022010_MSC140547280551273.EDR" is not an allocated inode.
Patching will delete the directory entry.
Remove directory entry? y
Directory entry "GSM_20080821022510_MSC140547280641273.EDR" is not an allocated inode.
Patching will delete the directory entry.
Remove directory entry? y
 
Checking log files
Checking extended attributes file
Checking allocation summary file
Checking policy file
Checking filesets metadata
Checking file reference counts

File inode 1191665 is not referenced by any directory.
Reattach inode to lost+found? y

File inode 1196881 is not referenced by any directory.
Reattach inode to lost+found? y

And it gave the result similar as above. However it has removed all the files who have bad entries.

But the corruption issues was because of the old version of IBM servers so their typical workaround is to do patching , and its 2 years old, so lot of patching is require and will be blind in production servers. So anyway cant go for it (not recommended).

Kindly let me know if any is familiar about any alternate on this, the details are as below :


Code:
<username@hostname>  $ uname -a
AIX <hostname> 3 5 00C0B4204C00

Kindly let me know if something else is required.

Moderator's Comments:
Mod Comment i have removed the excessive formatting and added CODE-tags instead. Please do so yourself henceforth.

Last edited by bakunin; 08-18-2010 at 06:31 AM..
# 2  
Old 08-19-2010
Hi,
Try:
Code:
mmstartup

now: ls and other commands.
# 3  
Old 09-06-2010
Hi, mmstartup is just to reboot the file system , my issue is little different , please go through the previous post.
# 4  
Old 09-06-2010
if fsck output contains only two types of errors as listed below, then probably all data blocks are intact and there is no data loss. but we'll need to have a look at the entire fsck output to be sure.

error type 1 : Directory entry ... is not an allocated inode
error type 2 : File inode 1191665 is not referenced by any directory

we need to check if these errors are related to each other. if they are, then all the inodes moved to lost+found are those whose directory entries were removed by fsck. in this case, you could check the contents of lost+found and if you could identify the entries in it, they could be restored.

in the meanwhile, i'm checking if the fsck expert person (about gpfs) could see this and tell us more.

edit : i checked with fsck expert who is working with gpfs and he confirmed that there should be no data loss (as stated above), and the detached entries should be available in lost+found which could be used for corrections.

Last edited by Yogesh Sawant; 09-06-2010 at 04:01 AM..
# 5  
Old 09-06-2010
You very lucky man, that have two weeks to repair filesystem.
Retrieve directory from archive.
# 6  
Old 09-07-2010
Quote:
Originally Posted by john1212
Retrieve directory from archive.
I'd even suggest recreating the filesystem completely anew before. Whatever has gone wrong in this FS should not be allowed to go on any further.

Create the filesystem(s) anew on new LUNs, restore its contents from backup and remount. This should account for only a very short change, if you plan your activities carefully.

I hope this helps.

bakunin
# 7  
Old 09-20-2010
mount the filesytem
test, have you any snapshots:
Code:
mmlssnapshots /dev/gpfslv -d

example:
Code:
Snapshots in file system fs1: [data and metadata in KB]
Directory  SnapId    Status     Created               Data  Metadata
snap1       1        Valid   Fri Oct 17 10:56:22 2003   0       512

create directory .problem and restore filesystem
Code:
mmsnapdir /dev/gpfslv -s .problem
mmrestorefs /dev/gpfslv snap1

you look for your missed directory and other in directory .problem
after them you can do:
Code:
rm -rf .problem

I hope this helps.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

GPFS NodeData File problem

Hi Everyone I have configured two node GPFS cluster and it is running fine. But today when I have rebooted one of the node and after starting GPFS one node shows down. on investigating I have checked that mmfsNodeData file is missing from location /var/mmfs/gen/ on one node. I am receiving the... (1 Reply)
Discussion started by: m_raheelahmed
1 Replies

2. Red Hat

Issue with mounting the file system .

Hi gurus, Need your help , Am facing some issue with one of the ext3 file system.while rebooting it failed to mount fs after running fsck i tried manuall but still no luck.However i made the machine up but am unable to mount file system can some one please help me. root@vm258902]~# cat... (2 Replies)
Discussion started by: kapil514
2 Replies

3. Shell Programming and Scripting

Shared File system- lib access issue

I have a C++ binary executable installed in a file system which is shared across multiple solaris boxes. When I start this executable from one of the boxes,I am able to start only 4 parallel instances and from the 5th instance onwards I am getting the following error. fatal: libdb2.so.1:... (2 Replies)
Discussion started by: prasperl
2 Replies

4. AIX

How to configure new hdisk such that it has a gpfs fs on it and is added to a running gpfs cluster?

Hi, I have a running GPFS cluster. For one every mountpoint that i have created i have one disk assigned to it. That disk is converted to an NSD and is a part of the GPFS Cluster. Now i have a new disk and there is this requirement to add it to the GPFS cluster, such that this becomes an NSD.... (1 Reply)
Discussion started by: aixromeo
1 Replies

5. AIX

two gpfs in one node issue

dear all can i create two gpfs in one node each gpfs is pointing to single hdiskpower (1 Reply)
Discussion started by: thecobra151
1 Replies

6. Solaris

Recovering from corrupt /etc/system file

Hi I would like to know one thing while recovering from corrupt /etc/system file. For that we have to do at ok> boot -a when system ask for system file we give /dev/null. Could any one tell me what /dev/null value will do in this case. Regards (1 Reply)
Discussion started by: sb200
1 Replies

7. AIX

Backup for GPFS file system

Unix people.. Backup for GPFS file system /dev/gpfs /application I have GPFS file system and I'd like to take backup for that file system to the tape I'm using this command Smitty fs + Backup a File System And I'm take the backup but this is will work if that gfs2 but I'm... (4 Replies)
Discussion started by: Mr.AIX
4 Replies

8. Solaris

/ file system full issue

Hi All, This is Babu working as a system administrator. Here I am getting one problem with one of my Sun server's root (/) file system. In df -h command / file system showing 7.8 GB used space.But in du -hd command it showing 5.2 gb only. Please can any one help me resolve this issue... (2 Replies)
Discussion started by: lbreddy
2 Replies

9. Solaris

root file system issue

Hi, In df -h root file system showing (total size) (used) (free) /dev/dsk/c0t0d0s0 11G 7.6G 2.8G 73% / but du -hd command showng 5.1 gb used sapce. I am missing 3.00 gb space. Here I have to knoe where is Remaining space... (2 Replies)
Discussion started by: lbreddy
2 Replies

10. Solaris

File system Issue

Hi All, I am working with the Solaris 9. In df -h command / file system size is 9.6 GB , used 7.5 GB and avialble space is 2.1. OK But in du -hd command it is showing some 4.1 GB only. How to find out that remaining 3.4GB. Can you please help me ... Thanks and Regards, (1 Reply)
Discussion started by: lbreddy
1 Replies
Login or Register to Ask a Question