Login or Register to Ask a Question and Join Our Community


AIX

Exportvg/importvg causes corrupt LV Control Block

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems AIX Exportvg/importvg causes corrupt LV Control Block
# 1  
Old 08-30-2013
Exportvg/importvg causes corrupt LV Control Block
Hi experts,

Power7 p720
AIX 6.1

This is what happened:

Code:
$ sudo importvg -y v7000_1vg hdisk6
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
v7000_1vg

Volume Group shows 'active'

Code:
$ lsvg v7000_1vg
VOLUME GROUP:       v7000_1vg                VG IDENTIFIER:  00f7974500004c0000000137ecf731f1
VG STATE:           active                  PP SIZE:        1024 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      3904 (3997696 megabytes)
MAX LVs:            256                      FREE PPs:       2487 (2546688 megabytes)
LVs:                21                       USED PPs:       1417 (1451008 megabytes)
OPEN LVs:           0                        QUORUM:         2 (Enabled)
TOTAL PVs:          1                        VG DESCRIPTORS: 2
STALE PVs:          0                        STALE PPs:      0
ACTIVE PVs:         1                        AUTO ON:        yes
MAX PPs per VG:     32512                                     
MAX PPs per PV:     4064                     MAX PVs:        8
LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      non-relocatable
PV RESTRICTION:     none                     INFINITE RETRY: no

Below, the fields 'TYPE' and 'MOUNT POINT' are missing

Code:
$ lsvg -l v7000_1vg
v7000_1vg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv02                        1       1       1    closed/syncd  N/A
fslv18                         11      11      1    closed/syncd  N/A
fslv19                         200     200     1    closed/syncd  N/A
fslv20                         100     100     1    closed/syncd  N/A
fslv21                         100     100     1    closed/syncd  N/A
fslv22                         130     130     1    closed/syncd  N/A
fslv23                         100     100     1    closed/syncd  N/A
fslv24                         100     100     1    closed/syncd  N/A
fslv25                         100     100     1    closed/syncd  N/A
fslv26                         100     100     1    closed/syncd  N/A
fslv27                         100     100     1    closed/syncd  N/A
fslv28                         100     100     1    closed/syncd  N/A
fslv29                         100     100     1    closed/syncd  N/A
fslv30                         30      30      1    closed/syncd  N/A
fslv31                         30      30      1    closed/syncd  N/A
fslv32                         30      30      1    closed/syncd  N/A
fslv33                         30      30      1    closed/syncd  N/A
fslv34                         30      30      1    closed/syncd  N/A
fslv35                         10      10      1    closed/syncd  N/A
fslv36                         10      10      1    closed/syncd  N/A
fslv37                         5       5       1    closed/syncd  N/A

I tried to mount manually

Code:
$ sudo mount /dev/fslv19 /notes/d24aml06/mail1
mount: 0506-322 Cannot determine log device to use for /dev/fslv19 (/notes/d24aml06/mail1).

Log LV is missing data

Code:
$ sudo getlvcb -AT loglv02
         
         intrapolicy =  
         copies = 1 
         interpolicy =  
         lvid =  
         lvname =  
         label =  
         machine id =  
         number lps = 1 
         relocatable =  
         strict =  
         stripe width = 0 
         stripe size in exponent = 0 
         type =  
         upperbound = 0 
         fs =  
         time created  =         time modified =

Below is the evidence that the Logical Volume Control Block is corrupted

Code:
$ sudo getlvcb -AT fslv19 
         B�
         intrapolicy = � 
         copies = 1 
         interpolicy = � 
         lvid = � 
         lvname =  
         label = ���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������� 
         machine id = ����������������� 
         number lps = 200 
         relocatable = � 
         strict = � 
         stripe width = 0 
         stripe size in exponent = 0 
         type =  
         upperbound = -21846 
         fs = �������������������������������������������������������������������������������������������������������������������������������� 
         time created  = �����������������������������������������������������������������������������   time modified = �����������������������������������������������

Unable to rebuild the LVCB

Code:
$ sudo synclvodm v7000_1vg
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.

In a nutshell, after importing the VG I cannot use the data within it.

Next step?
# 2  
Old 08-30-2013
can you post the error report.
Code:
errpt

is this internal disk or coming through SAN
# 3  
Old 08-30-2013
Quote:
Originally Posted by filosophizer
can you post the error report.
Code:
errpt

is this internal disk or coming through SAN
This is SAN

Code:
$ lsdev -Cc disk | grep hdisk6
hdisk6 Available 03-00-02 IBM MPIO DS5020 Array Disk

Code:
$ lspath -l hdisk6
Enabled hdisk6 fscsi0
Enabled hdisk6 fscsi2

Code:
$ errpt
B6267342   0828123013 P H hdisk6         DISK OPERATION ERROR
C1348779   0828123013 I O SYSJ2          LOG I/O ERROR
E86653C3   0828123013 P H LVDD           I/O ERROR DETECTED BY LVM
B6267342   0828123013 P H hdisk6         DISK OPERATION ERROR
C1348779   0828123013 I O SYSJ2          LOG I/O ERROR
E86653C3   0828123013 P H LVDD           I/O ERROR DETECTED BY LVM
B6267342   0828123013 P H hdisk6         DISK OPERATION ERROR
C1348779   0828123013 I O SYSJ2          LOG I/O ERROR
E86653C3   0828123013 P H LVDD           I/O ERROR DETECTED BY LVM

---------- Post updated at 05:33 PM ---------- Previous update was at 05:05 PM ----------

Tried to update the LVCB manually...

Code:
$ sudo chlv -t jfs2log loglv02
0516-711 chlv: Warning, unable to update logical volume
        control block.

no can do Smilie
Last edited by livehho; 08-30-2013 at 08:12 PM..
# 4  
Old 08-31-2013
A wild guess is that some locks remained when you exported the VG. Is the VG in ECM (enhanced concurrent mode)?

If so, try a "varyonvg -b -u", then "exportvg", then reimport again. This might help.

I hope this helps.

bakunin
# 5  
Old 08-31-2013
Quote:
Originally Posted by bakunin
A wild guess is that some locks remained when you exported the VG. Is the VG in ECM (enhanced concurrent mode)?

If so, try a "varyonvg -b -u", then "exportvg", then reimport again. This might help.

I hope this helps.

bakunin
nope.. it's not concurrent mode
# 6  
Old 08-31-2013
The next idea is checking the VGDA, it might just be corrupted.

If this is the case, you might have to repair the VGDA data, which is not an easy task (and without danger neither).

There is an (undocumented! so use at your own risk) command "readvgda". If used on a good copy it will show you the complete VG layout, including LVs, their PP mappings, etc.. I suggest you read also the LVM redbook to understand the concepts behind the following actions. CAUTION: these actions have the potential to harm your data, carefully examine all the steps, save everything you do and only proceed when feeling comfortable!

You need a "/etc/filesystems" too, because you do not get the mount points from the VGDA.

Now construct map files for each LV from the VGDA info. Then export the VG and recreate it on the original PVs and with the original PP size, then recreate each LV usin the constructed map file on its original place:

Code:
mkvg -f -y <VGname> -s <PPsize> <hdisk_dev_1> <hdisk_dev_2> ...
mklv -y <LVname> -m /path/to/mapfile -t jfslog VGname <numlp>
mklv ...

(I was supposing jfs here, change the commands accordingly if you use jfs2.)

Now restore the "/etc/filesystems" but DO NOT MOUNT yet!

For each filesystem update the LVCB with the log and label information:

Code:
chfs -a log=<logdevice> <mountpoint>
chlv -L <mountpoint> <LVname>

NOW run "fsck" over each FS - WITHOUT the "-y"! If "fsck" complains about not being able to read the superblock the respective LV was not recreated correctly and you have to start over. If the jfslog LV was not recreated correctly, then "fsck" will not be able to replay the log and you will probably lose some data, so better double- and triple-check your mappings before you answer "yes" to any of "fsck" questions if it should correct anything.

Note, that if a filesystem was online at the time of the crash fsck might find some (unrecoverable) errors because of corrupted inode-lists. The only thing you can do is to fix this, but it will probably make you lose data. (Serves you right if you have no working backup.)

Re-run "fsck" until there are no errors any more to correct, then mount FSes. If any errors were fixed you should check "lost+found" for the remains of orphaned files/blocks. You might be able to regain some of the lost data from there.

I hope this helps.

bakunin
Last edited by bakunin; 09-02-2013 at 01:49 PM..
This User Gave Thanks to bakunin For This Post:
livehho
# 7  
Old 09-02-2013
I don't have the original /etc/filesystems

I am contemplating recreating the VG (loosing all data)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. High Performance Computing

Postgresql Database Corrupt

I am managing a linux cluster which has been build on Platform Cluster Manager PCM 1.2.1) from IBM Platform Computing. Unfortunately somebody deteled data files of postgresql from /var/lib directory. I somehow managed to start the postmaster service again, but all the administrative commands of... (2 Replies)
Discussion started by: ahsanpmd
2 Replies

2. AIX

importvg question

Is there a way to find the volume group name after assigning hdisk to the machine for importing it ? importvg -y <find_the_name_from_the_unknown_disk_> hdiskX Code tags and no hijacking. You should be aware of that ;) (3 Replies)
Discussion started by: filosophizer
3 Replies

3. Shell Programming and Scripting

cp to copy only non-corrupt files

I don't know if I am asking this correctly, but I have a hard drive with some bad sectors and it appears that some of the data is corrupt. I am having allot of trouble copying the data to a new drive. The issue is not in copying files, but that the new drive to which files are copied is not acting... (17 Replies)
Discussion started by: LMHmedchem
17 Replies

4. HP-UX

corrupt disk

Hallo Friends, I have application X running on hpux 11.11 and oracle 9i release 2. I recently had a hardware failure on disk /dev/dsk/c2t0d0 Below is the systemlog file : root@a7dmc:/var/adm/syslog > /opt/resmon/bin/resdata -R 155713541 -r /storage/events/enclosures/gazemon/0_1_1_0.0.0... (11 Replies)
Discussion started by: kekanap
11 Replies

5. AIX

exportvg

Hi How to check if a vg is exported from an LPAR using exportvg? Is there a command or place where we can see the list of the vg's exported from the LPAR ? Because if i run the command to varyonvg i get the below error : root@cbspsrc02 #varyonvg old_rootvg 0516-082 varyonvg: Unable to... (7 Replies)
Discussion started by: newtoaixos
7 Replies

6. AIX

Unable to execute varyonvg during importvg

Hello, On Aix 5.3, during importvg, the varyonvg fails: importvg -y vgtest hdisk20 0516-013 varyonvg: The volume group cannot be varied on because there are no good copies of the descriptor area. When i use manually the command varyonvg -u -b -t vgtest to force, the vg can... (3 Replies)
Discussion started by: dantares
3 Replies

7. Solaris

How to corrupt a superblock?

I need to corrupt a superblock of a mounted device in a soalris m/c and check recovery from an alternate superblock. How can this be done? (2 Replies)
Discussion started by: sujathan
2 Replies

8. AIX

nooB questions on importvg and exportvg

I don't understand the concept behind importvg and exportvg. What are some examples of when one would use this? (1 Reply)
Discussion started by: outtacontrol
1 Replies

9. UNIX for Dummies Questions & Answers

Causing a disk to be corrupt

Hmm, how to ask this without sounding too malicious... How might one go about causing a disk corruption in OS X specifically or via the command line in UNIX in general? Doesnt matter the severity of the problem, I just want to scare the person a little, then fix the problem for them. Any... (1 Reply)
Discussion started by: Yummator
1 Replies

10. Programming

Reading a process control block

Hello, I want to know what call to use to read the details of a process control block in solaris ?:) (2 Replies)
Discussion started by: hmurali
2 Replies
Login or Register to Ask a Question