Stale PPs in AIX, failed disks.. how to replace?


 
Thread Tools Search this Thread
Operating Systems AIX Stale PPs in AIX, failed disks.. how to replace?
# 8  
Old 10-28-2014
First off, your "inter-policy" is set to "maximum", which is nonsense. "inter-policy=maximum" means to use as many PVs as possible when creating or extending a LV, which results in a "poor mans striping". The basic idea is to distribute I/O over as many disks as possible. This might make sense with very small PP sizes (2-16MB) and many very old disks. With todays disks and todays typical PP sizes it is utter nonsense and only makes administration overly complicated. My suggestion is to get rid of it. Use an inter-policy of "minimum" only.

Next: your VG is 1.5 TB in size and has a laughable amount of 128M in contingency! I suggest you get some more disk space to allow at least for reorganisations, which should take place regularly when you deal with physical disks. Here, like in any area, it doesn't pay off to be greedy and it certainly lowers efficiency when using a resource to its abolute limit. You do not use your processor ressources at a permanent 100%, you do not tune your system to death so that it constantly has zero free RAM and for the same reason you do not fill your disks to absolute capacity - there is no room left to react if something unforeseen comes up. Plus some VG operations need unused space to be undertaken. You should have at least 100-200GB free space once you have created all the filesystems, better even more.

My suggestion is to recreate the VG completely anew: backup your data using the savevg utility, delete the VG (you can use exportvg to remove it completely from the system), then recreate it using the restvg. Select sensible values for the PP size when recreating it, because your system seems to use pretty small disks: you have 22 (now 21) PVs in your VG and a net capacity of 1.5TB, the single average disk is about 70G in size. No disk you can get today has as few as 70GB but with your PP size of 128M you can't use any disk larger than ~128G. On the other hand you cannot use a factor as i suggested above, because even the smallest factor of 2 would limit the VG to 16 PVs which you already exceed. Therefore, when recreating the VG use a PP size of 512MB or even bigger, which will allow you to replace the (probably quite old) disks with new, bigger ones. Be warned that this operation will take some time and you need downtime to do it.

If you can get additional disks to hold the data you could just create a new VG and copy the data from the old one, then delete the old one, this way you save the time for creating the backup and you always have a working copy as a fallback solution. A healthy leayout would be 4 PVs of 500GB each, which gives you 2TB of space, 1.5 TB taken and 500GB as a reserve to draw on. This (or the solution above) is the recommended way to do it.

The next best is to reorganize the existing VG on the existing disks. Some problems described above will stay, but at least the problems internal to the VG could be repaired.

Reorganisation means: remove all the mirrors from all the LVs in the VG. The command is unmirrorvg. This way you get some free space to work with. Then change the LVs using the chlv command to set inter-policy to "minimum". Once you have done this use the reorgvg command to bring the new LV layout into place. Be prepared to wait for quite some time (probably some hours) before this is finished, it will physically relocate the LPs of all the LVs. You can do all this online without downtime, but i suggest to do it in times of least activity or even take a downtime anyway. It will certainly be slowed down by much system stress and users will notice the degraded performance when this operation runs. Finally remirror using the mklvcopy command or - if you can get additional disks to add to the VG - a mirrorvg which does a mklvcopy for all LVs.

If you do a mirrorvg here is a tip: switch off the syncing, because it uses only 1 PP at a time. Do it this way (deferred syncing):

Code:
# mirrorvg -s myvg
# syncvg -P <NN> -v myvg

For "NN" set a number 1-32, which is the number of PPs be synced in parallel. Which number is good depends on your RAM, load, PP size and other things, but i suggest 10-20 for PP sizes between 64M and 512M.

I hope this helps.

bakunin

/PS:

Quote:
Interesting error, since hdisk42 shows it is not assigned when I ssue "lspv hdisk42"
This might be because a VGDA (volume group descriptor area) was found on the disk. Maybe it was already in use in this or another system and already held data. In any way, when you forced it into the VG it was cleaned of these data and reformatted, so speculations about its former content are moot at this point.

bakunin

Last edited by bakunin; 10-28-2014 at 05:24 PM..
# 9  
Old 10-29-2014
Thanks for the reply.

I am following your 2nd suggestion as this machine is going to be out its door in about 2 months (its over 5+ years old) and is a DR/testing box so not a production machine, less critical. My goal now is to keep it going for the next 2 months.

My goal is to get /usr1 (vg_usr1) working, removing the failed disks and making sure it is mirrored.

Sorry, perhaps I should have stated this first though your notes I am going to use on the new system we are putting in place.

I changed the policy to minimum, and tried to run reorgvg...

Code:
# lsvg vg_usr1
VOLUME GROUP:       vg_usr1                  VG IDENTIFIER:  000015010000d60000000141138f2577
VG STATE:           active                   PP SIZE:        128 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      11466 (1467648 megabytes)
MAX LVs:            256                      FREE PPs:       547 (70016 megabytes)
LVs:                2                        USED PPs:       10919 (1397632 megabytes)
OPEN LVs:           0                        QUORUM:         11 (Enabled)
TOTAL PVs:          21                       VG DESCRIPTORS: 21
STALE PVs:          0                        STALE PPs:      0
ACTIVE PVs:         21                       AUTO ON:        yes
MAX PPs per VG:     32512
MAX PPs per PV:     1016                     MAX PVs:        32
LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable
PV RESTRICTION:     none                     INFINITE RETRY: no
DISK BLOCK SIZE:    512

# lslv lv_usr1
LOGICAL VOLUME:     lv_usr1                VOLUME GROUP:   vg_usr1
LV IDENTIFIER:      000015010000d60000000141138f2577.1 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       closed/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            32512                  PP SIZE:        128 megabyte(s)
COPIES:             2                      SCHED POLICY:   parallel
LPs:                6005                   PPs:            12010
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        /usr1                  LABEL:          /usr1
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO
INFINITE RETRY:     no

Code:
# reorgvg vg_usr1
0516-1939 lmigratepp: PV identifier not found in VGDA.
0516-964 reorgvg: Unable to migrate logical volume lv_usr1.
0516-968 reorgvg: Unable to reorganize volume group.
0516-962 reorgvg: Logical volume loglv00 migrated.

Sounds like one of the pvs (hard drives) could be having an issue that is currently assigned to the vg?
# 10  
Old 10-29-2014
Quote:
Originally Posted by c3rb3rus
I am following your 2nd suggestion as this machine is going to be out its door in about 2 months (its over 5+ years old) and is a DR/testing box so not a production machine, less critical. My goal now is to keep it going for the next 2 months.
OK, point taken.


Quote:
Originally Posted by c3rb3rus
I changed the policy to minimum, and tried to run reorgvg...
You may have read over it: i suggested to completely unmirror all the LVs (in the VG) first. From your lslv output i can see that at least "lv_usr1" is still mirrored - and most probably some mirror copies of certain LPs are already gone with the missing disks. I suppose the other LVs in the volume are mirrored too and their mirroring status is uncertain in the same way. Furthermore it makes no sense to reorganize a mirrored volume instead of unmirroring it, reorganize it and at last remirror it again. It simply doubles the work and limits the reorganization options.

Quote:
Originally Posted by c3rb3rus
Sounds like one of the pvs (hard drives) could be having an issue that is currently assigned to the vg?
No, not necessarily. A possible reason is that some PPs with mirror copies are not where the system expects them to be because the disks which used to provide these PPs are gone. Finally: you replaced two missing disks by one new one and most probably you lost raw diskspace. the VG was filled to capacity at the start and if you further reduced the available raw space you cannot mirror all the LVs any more.

Therefore:

- unmirror the VG completely. If you worry about possible data loss take a backup before but unmirror anyway. Do exhaustive fscks before to make sure the FSes are OK. (You will need downtimes for the fsck because the FSes must be unmounted.)

- After this is finished take stock of: how much raw space do you have and how much is used. If the space in your VG is already 50% or more used a complete mirror is not possible and you will have to select certain FSes/LVs which will get mirrors and others which don't. One way to circumvent this is to reduce the size of LVs/FSes to get usage below 50% again. Maybe some filesystems could do with a little less space?

If so, here is how to it: this command will reduce a FS "/some/where" and its underlying LV by 1GB. Notice that you can only assign whole PPs to a LV, therefore shrinking/expanding only takes place in steps of the PP size.

Code:
chfs -a "size=-1G" /some/where

- now do a reorgvg.

- set up mirrors again using the "mirrorvg" command. Notice that you might need to move a few LVs a second time because you have an odd number of PVs and mirrors are created on different disks (see your lslv output). You do this with the "migratepv" command, with which you can move around one or all the PPs of a LV to certain positions on certain PVs.

It might take considerably less time to get a second replacement disk and make the number of available PVs even again.

I hope this helps.

bakunin
# 11  
Old 11-20-2014
Hey bakunin Smilie

I decided to destroy one of my volume groups (it was used for testing) where we had the multiple disk failures to address rootvg being stale..

I found this article and followed it to fix my (rootvg.

Here are the events..

Code:
# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk10           active            546         497         109..89..81..109..109
hdisk11           active            546         173         110..56..00..00..07
hdisk12           active            546         177         68..00..00..00..109
hdisk13           active            546         487         110..50..109..109..109
hdisk14           missing           546         511         109..89..95..109..109
hdisk15           active            546         501         110..109..64..109..109

# unmirrorvg rootvg hdisk14
0516-1246 rmlvcopy: If hd5 is the boot logical volume, please run 'chpv -c <diskname>'
        as root user to clear the boot record and avoid a potential boot
        off an old boot image that may reside on the disk from which this
        logical volume is moved/removed.

0301-108 mkboot: Unable to read file blocks. Return code: -1
0516-1144 unmirrorvg: rootvg successfully unmirrored, user should perform
        bosboot of system to reinitialize boot records.  Then, user must modify
        bootlist to just include:  hdisk10.

# rmdev -l hdisk14 -d
hdisk14 deleted

# bootinfo -b
hdisk10

# Do a chdev on hdisk1 to remove the pvid: 
chdev -l hdisk1 -a pv=clear

# extendvg rootvg hdisk20
0516-1254 extendvg: Changing the PVID in the ODM.
0516-1398 extendvg: The physical volume hdisk20, appears to belong to
another volume group. Use the force option to add this physical volume
to a volume group.
0516-792 extendvg: Unable to extend volume group.

# extendvg -f rootvg hdisk20

# mirrorvg rootvg
0516-1804 chvg: The quorum change takes effect immediately.
0516-1126 mirrorvg: rootvg successfully mirrored, user should perform
        bosboot of system to initialize boot records.  Then, user must modify
        bootlist to include:  hdisk11 hdisk10.

# # lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk10           active            546         423         109..34..62..109..109
hdisk11           active            546         152         109..36..00..00..07
hdisk12           active            546         177         68..00..00..00..109
hdisk13           active            546         485         110..50..107..109..109
hdisk20           active            546         546         110..109..109..109..109
hdisk15           active            546         501         110..109..64..109..109

# lsvg -l rootvg
rootvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
hd5                 boot       1       2       2    closed/syncd  N/A
hd6                 paging     4       8       2    open/syncd    N/A
hd8                 jfs2log    1       2       2    open/syncd    N/A
hd4                 jfs2       2       4       2    open/syncd    /
hd2                 jfs2       26      52      2    open/syncd    /usr
hd9var              jfs2       9       18      2    open/syncd    /var
hd3                 jfs2       2       4       2    open/syncd    /tmp
hd1                 jfs2       320     640     2    open/syncd    /home
hd10opt             jfs2       7       14      2    open/syncd    /opt
hd11admin           jfs2       1       2       2    open/syncd    /admin
fwdump              jfs2       8       16      2    open/syncd    /var/adm/ras/platform
lg_dumplv           sysdump    48      96      2    open/syncd    N/A
livedump            jfs2       8       16      2    open/syncd    /var/adm/ras/livedump
paging00            paging     4       8       2    open/syncd    N/A
lv_inst             jfs2       55      110     2    closed/syncd  N/A

# bootinfo -b
hdisk10
# bootlist -m normal -o
hdisk10 blv=hd5 pathid=0
-
# bosboot -a

bosboot: Boot image is 55324 512 byte blocks.
# bootlist -m normal -o
hdisk10 blv=hd5 pathid=0
-

Now I did see something strange.. I have not yet phisically removed the failed disk (hdisk14) so when I run cfgmgr it gets picked up..

Code:
# lsdev -Cc disk | grep "hdisk14"
hdisk14 Available 03-08-01-5,0  16 Bit LVD SCSI Disk Drive
# lspv | grep "hdisk14"
hdisk14         none                                None

In turn, if I now run "bootlist -m normal -o" I get..

Code:
hdisk10 blv=hd5 pathid=0
hdisk14 pathid=0

Why is it showing there? I am concerned if a reboot happens I may not be able to boot.. how can I remove it and assign hdisk20 which is the replacement for hdisk14 in rootvg... or do I need to do anything further at this point?

Maybe just rmdev -l hdisk14 -d and leave it?

---------- Post updated 11-20-14 at 02:12 PM ---------- Previous update was 11-19-14 at 06:58 PM ----------

I am going to re-create the vg/lv and start from scratch.
# 12  
Old 11-21-2014
Quote:
Originally Posted by c3rb3rus
In turn, if I now run "bootlist -m normal -o" I get..

Code:
hdisk10 blv=hd5 pathid=0
hdisk14 pathid=0

Why is it showing there? I am concerned if a reboot happens I may not be able to boot.. how can I remove it and assign hdisk20 which is the replacement for hdisk14 in rootvg... or do I need to do anything further at this point?
There are two commands you need to know: "bosboot" and "bootlist".

With "bootlist" you can set (or display) the order (in NVRAM) in which devices are used to boot from. There are basically 2 lists of such boot devices, "normal" and "service". There is a "key position" that determines which boot list is getting used. Back then, in the times of physical machines, this was really a lock with a key and the key could be in one of 3 possible positions, "normal", "service" (sometimes named "maint") and "lock". Today this is only a virtual construct, but the terminology stuck.

With "bosboot" you bring a boot image to boot from onto a device. This copies the code that is executed during the boot onto the device you designated as boot device with "bootlist".

I suggest you read carefully the man pages of both commands, then read them again. You can render the system unusable with these commands, so spend some time and effort to really, firmly understand what they are doing. Only then use them to alter the boot list. (You will need both of them - proceed only if you understand why.)

I hope this helps.

bakunin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

DISKs are gone after shutting down and replace an FC card

Hi there, I had issue with one of MY FC cards on T4-2 servers so system team replace it and start the machine but when launch FORMAT command so I don't see my shared disks coming from storage controller. i have checked at the Fabric switch so WWN numbers are visible and zones are ok and after... (1 Reply)
Discussion started by: janakors
1 Replies

2. AIX

AIX - stale partition

Hi everybody, I have a little problem with my AIX 6.1, PowerHA 6.1 LVM mirror. After problem with SAN pathing of our one Datacenter, I have LV at stale state. # lsvg cpsdata2vg VOLUME GROUP: cpsdata2vg VG IDENTIFIER: 00fb518c00004c0000000169445f4c2c VG STATE: ... (6 Replies)
Discussion started by: Necronomic
6 Replies

3. AIX

How to reclaim hard disks and IP's in AIX?

Hello I recently received a request to reclaim hard disks and IP addresses within an AIX system(s). THe file systems are no longer in use and the client has indicated that it is OK to remove them and reclaim the disks and release the IP's. Now, since the file systems belong to a Volume group I... (8 Replies)
Discussion started by: Joseph Sabo
8 Replies

4. AIX

Issue with increasing size of SAN disks on AIX

HI, I have had an issue last night while trying to extend a filesystsem . chvg -g <vg> command cameback with an error 0516-1790 chvg: Failed bootinfo -s hdisk9. Ensure the physical volume is available and try again. 0516-732 chvg: Unable to change volume group u01vg. the VG has 1... (1 Reply)
Discussion started by: xmen01
1 Replies

5. Solaris

Command to find the failed disks in SVM and VxVM

How to find & replace bad disks in solaris & VXVM?plz explain step by step? (2 Replies)
Discussion started by: xtreams
2 Replies

6. AIX

Configuring new disks on AIX cluster

We run two p5 nodes running AIX 5L in a cluster mode (HACMP), both the nodes share external disk arrays. Only the primary node can access the shared disks at a given point of time. We are in the process of adding two new disks to the disk arrays so as to make them available to the existing... (3 Replies)
Discussion started by: dnicky
3 Replies

7. AIX

Configurin EMC SAN disks on AIX

This may sound like an absolute rookie question, and it is. I have been working on Migrating our HP and Solaris servers to the new EMC SAN and know the routines backwards. Now we've suddenly got a new IBM server and I don't even know how to check if it is connected to the switch. Can someone... (1 Reply)
Discussion started by: ronellevan
1 Replies

8. AIX

system disks on aix 5.3

hello i'm running on P570 box aix 5.3 8 cpus 24G ram there are 1850 users loged in to this box the problem is that the two sysytem disks busy all the time hdisk0 100% busy hdisk1 100% busy some one have an idea what writing to this disks? thanks ariec (9 Replies)
Discussion started by: ariec
9 Replies

9. AIX

Replace a Failed Root Disk in AIX

My root disk is failed and how to replace the root disk in AIX. Can u give a detailed explanation in step wise. Pls give the answer taking different scenarios. Regards Praveen (1 Reply)
Discussion started by: chinnu_mulakala
1 Replies

10. UNIX for Dummies Questions & Answers

Disks on AIX

Hello I've been working on AIX 5.3 ML3 on IBM pSeries520. That server has 6 HDD drives in 3 volume groups (1+mirror in each group). I must check which phisical disk is which disk in the system. For ex. I want to know that disk in 4th slot in the machine is marked as hdisk5 on AIX. Does anybody... (2 Replies)
Discussion started by: piooooter
2 Replies
Login or Register to Ask a Question