Question about Grub2 and Multipathing

advanced, grub, grub2, ipa, multipath, question, san boot

Thread Tools Search this Thread
# 1  
Old 10-10-2018
Question about Grub2 and Multipathing

I have a bit of an odd situation I would like to float out here and see if anyone has any ideas on this..

We are working on doing Disaster Recovery on a number of RHEL 7.4 systems. These are running on Cisco Blade Servers.

The mount point for /boot is on a Multipathed SAN LUN. There are a couple of other SAN volumes as well in disk groups; none of those are being problematic, but /boot is.

/dev/mapper/360000970000197700328533030344233p1  976M  175M  751M  19% /boot

Above is the mount for the production system. We use SRDF to copy this data to a DR site and then clone the storage and boot up the same hardware at the DR site; Cisco blade servers (same models) as in production.

When we boot up the DR nodes, the system understandably gets a little 'confused' and will mount /boot to a RAW device.

/dev/sdy1                            976M  175M  751M  19% /boot

Note /dev/sd* instead of the MPATH device /dev/mapper/***

So to fix this:
First, I run a 'multipath -W' and this corrects the extra WWIDs in /etc/multipath/wwids file.

multipath -W
successfully reset wwids

Now the WWIDS from the production side are gone and only the new ones that are at the DR site exist - So that file is now a happy file. Smilie

After that, I need to add a filter to /etc/lvm/lvm.conf to ignore any devices aside from the /dev/mapper devices (RedHat support suggested that I add the global_filter as well - but it seemed to work ok with just 'filter', but it didn't hurt either..

filter = [ "a|/dev/mapper/.*|", "r|.*|" ]
global_filter = [ "a|/dev/mapper/.*|", "r|.*|" ]

Then - create a new initramfs image:

dracut --force --add multipath --include /etc/multipath

And reboot.
The server comes back up in either rescue or emergency mode (I'll pay more attention next time) and EACH TIME, running grub2-mkconfig fixes it - and the server boots just fine.

I need to figure out what's going on for my own geeky-obsessive-ness. The thing is, I saved a backup copy of /boot/grub2/grub.cfg and compared it to the new one that was generated in emergency mode and there are zero differences. I used notepad ++ and did a file comparison - even adding a character to verify the plug-in was working right and I can find no difference at all between the two files.

I thought that grub2-mkconfig just generated a new grub.cfg file, but it almost seems like something else is going on here as well.

Any ideas?

It's not that I can't get these servers back online, it's just that I would like to skip the reboot into rescue mode - as we are looking to automate this process as much as possible.

We have recovered these 4 nodes a couple of times - this process seems consistent. I just can't figure out what change grub2-mkconfig is making to the system to get it to boot!

Thanks in advance!
# 2  
Old 10-10-2018
I have no experience with boot from SAN on metal, to put a small disclaimer upfront.

Not related to the problem...what would really help you is to have a separated installed operating system on both environments (DCs), with data part replicated in separate volume groups.
That way, only thing you do is import the volume group and (possibly or not depends) an ip assignment, depending on the topology and services such as vrrp or higher layer used.

I never endorsed SAN boot on metal personally (but in vms of course), always had couple of local disks to mirror ...

Point being, i would go with service separation and clean install on each of the sites, while cloning data part via storage methods.

Hopefully someone else who used linux SAN boot will be of more assistance with the actual problem.

This User Gave Thanks to Peasant For This Post:
Overcast451 (10-10-2018)
# 3  
Old 10-10-2018
Some of that is outside of my control Smilie

This is the method that was decided upon... we have become pretty endeared to using recovered Virtual Machines or volumes from our Production side; overall it has reduced our RTO significantly due to the ease of importing synchronized data.

Part of the long standing issue we have had with maintaining the OS portion at the DR site is changes made in production that don't get replicated to the DR environment. Those changes are supposed to be done, but often still get missed. While there is always the 'so and so forgot to...' and then management whines about it; doing a full replication of even the boot environments simply factors out those human problems. For instance - a new mount point is added or an old mount point is removed.. a ton of little annoying things. Then when we test out DR processes, sometimes these 'little things' that we are unaware were changed, can cost us large amounts of time. Lately, we have been consolidating ZPOOLS on Solaris and every time we test - we run into issues with them.

But overall this works very well. Everything is 100% except for this /boot MPATH issue; and honestly - the systems will run on a single path just fine. But you know.. that's not good enough for a geek such as myself! lol

Thanks for your input too Smilie
# 4  
Old 10-11-2018
After some reading it looks like a bug
RHEL7: Booting fails for cloned SAN root disk on multipath systems - Red Hat Customer Portal

Can you check if the version mentioned on top is your version of dracut

# 5  
Old 10-11-2018
That might just be it... I did find that running grub is *entirely* irrelevant, as /boot isn't even mounted in rescue mode in my case.

I added filters to /etc/lvm/lvm.conf
Rebuilt the initramfs ( dracut --force --add multipath --include /etc/multipath )
Cleaned up the additional WWIDs using 'multipath -W'

Then rebooted - came up in rescue mode.
I did nothing - but reboot again, and it came up just fine.

I will check into this bug and workaround and see what happens. Thanks for the find Peasant Smilie

------ Post updated at 01:54 PM ------

Oh and the version is a bit higher..

But on article you linked, under the "affected systems" - the output is what I'm getting..

Either way, I went ahead and updated to dracut-033-552.el7 - just to see if there's any difference.

I think these are the related errors on boot:

------ Post updated at 01:56 PM ------

Only at 4 posts, so can't post a link - but if you take out the spaces, I uploaded a pic of the KVM screen with one of the errors I think are related.

Aukro and eBay auction gallery -

Last edited by Corona688; 10-11-2018 at 03:17 PM.. Reason: fixed link

Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Unable to edit grub2 boot screen in centos 7 sabsac UNIX for Beginners Questions & Answers 1 12-03-2016 03:47 PM
Verify multipathing keelba Red Hat 1 10-01-2014 10:33 AM
GRUB2 + UEFI issue, new entry each boot sea Red Hat 1 03-29-2014 08:12 PM
MPxIO Multipathing Problem jamba1 Solaris 2 03-10-2013 08:10 PM
Multipathing in AIX manoj.solaris AIX 4 08-24-2011 02:39 AM
How to Cleanup Multipathing Tirmazi Red Hat 2 06-22-2011 11:48 AM
grub2 startup freeze Luke Bonham Linux 0 05-10-2011 04:37 PM
Grub2 (dual boot, dmraid) cannot run Debian6 neutralTTY Debian 0 02-09-2011 03:48 PM
Can't get puppy to work with grub2 Narnie UNIX for Dummies Questions & Answers 2 05-04-2010 04:31 PM
Multipathing - problem bieszczaders Solaris 4 04-22-2010 02:05 AM
Solaris multipathing joshmani Solaris 2 04-19-2010 12:42 PM
Assigning a vertual IP and doing IP multipathing... prashantshukla IP Networking 2 08-25-2008 02:12 PM
Solaris IP Multipathing maadhuu Solaris 0 06-17-2008 03:10 AM
solaris multipathing simquest Solaris 4 03-20-2008 04:06 PM
Strange Multipathing problem tjlst15 Solaris 3 09-13-2005 07:31 PM