Multiple Paths to SAN with LVM in Linux?

deckard · #1 (**permalink**) 08-09-2007

I'm setting up a CentOS 5 server that will be connected to an HP EVA4000. My previous experience is with an HP VA7400 and HP-UX. In HP-UX I can add "alternate paths" to a volume group in order to have redundant paths to the SAN via dual fiber channel HBAs just by either adding them with 'vgextend' or appending the appropriate PV paths when using 'vgcreate'. I tried doing something similar in CentOS with 'vgextend' and got some errors about duplicate PV ids. The LUN (a single one) I create on the EVA4000 show up on the CentOS box as /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd. And when I partitioned /dev/sda so that I had /dev/sda1, I also saw that /dev/sdb1, /dev/sdc1 and /dev/sdd1 were automagically there. So I know that these extra devices reflect the dual fiber channel HBAs and the dual paths through our redundant fiber switches.

My question is... did I do this the right way? When I do a 'vgdisplay' now I get:

Code:

[root@localhost ~]# vgdisplay
  Found duplicate PV Up921bgzWK4I7xtw7JiOB3VrdbVZhH2Z: using /dev/disk/by-path/pci-0000:13:00.0-fc-0x50001fe1500ca568:0x0001000000000000-part1 not /dev/sda1
  Found duplicate PV Up921bgzWK4I7xtw7JiOB3VrdbVZhH2Z: using /dev/disk/by-id/scsi-3600508b400070ac40000900000090000-part1 not /dev/disk/by-path/pci-0000:13:00.0-fc-0x50001fe1500ca568:0x0001000000000000-part1

at the beginning of the expected output. I want to avoid having to test the redundancy by disconnecting the fiber if possible. And in all honesty I don't think this is the right way to set up redundant paths on CentOS (or any other Linux distro for that matter). I think the "alternate paths" functionality is specific to HP-UX's implementation of LVM. But I suspect there is a "right way" to do alternate paths on Linux for redundancy.

thebap · #2 (**permalink**) 08-09-2007

We've just completed a project using RedHat Ent Linux and an HP MSA1500 SAN.

Running the controllers in active/active mode we're using RedHats Device Mapper Multipath software to create multiple paths to our SAN.

I suspect the CentOS will have similar capabilities and be able to use Redhat RPMs for DM/DMMP.

Oh and the DM/DMMP technology is built on LVM so in a way you're still using LVM, I hope this helps you out a little.

deckard · #3 (**permalink**) 08-09-2007

Quote:

Originally Posted by thebap

...Running the controllers in active/active mode we're using RedHats Device Mapper Multipath software to create multiple paths to our SAN... I hope this helps you out a little...

It sure did! Since CentOS uses the 'yum' system some searching on the web and within the yum respositories led me to this:

Code:

[root@localhost etc]# yum install device-mapper-multipath
Loading "installonlyn" plugin
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for device-mapper-multipath to pack into transaction set.
device-mapper-multipath-0 100% |=========================| 9.5 kB    00:00     
---> Package device-mapper-multipath.i386 0:0.4.7-8.el5 set to be updated
--> Running transaction check

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size 
=============================================================================
Installing:
 device-mapper-multipath  i386       0.4.7-8.el5      base              760 k

Transaction Summary
=============================================================================
Install      1 Package(s)         
Update       0 Package(s)         
Remove       0 Package(s)         

Total download size: 760 k
Is this ok [y/N]: y
Downloading Packages:
(1/1): device-mapper-mult 100% |=========================| 760 kB    00:01     
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: device-mapper-multipath      ######################### [1/1] 

Installed: device-mapper-multipath.i386 0:0.4.7-8.el5
Complete!

Now I just need to find a good HOWTO and give this a shot. Thanks a LOT!

deckard · #4 (**permalink**) 08-10-2007

Here's what I've come up with from start to finish (OS install through finally setting up multipath using the RAID subsystem) and I will be testing it from the ground up today. I mistakenly installed the 32-bit version of CentOS during my testing phase, so I am starting over with 64-bit today. If I run into any snags, I'll probably post back here. This is the procedure:

1. Install CentOS 5 (64-bit) with Xen Kernel
2. Install 'screen' (My personal preference for detached terminal sessions. For those who don't know, think of screen as "VNC" for the command line in terms of coming back to a terminal that you detached from earlier.), 'gcc', 'kernel-xen-devel-2.6.18-8.el5' using 'yum':

Code:

    yum install screen
    yum install gcc
    yum install kernel-xen-devel-2.6.18-8.el5

3. Install Emulex 'lpfc' driver to support the fiber channel cards. There is a simple install script in the driver tarball that woks if you have a compiler and kernel sources for your running kernel. The driver that comes with the OS doesn't appear to support the HBAnyware software mentioned in the next item which was the only way I was able to discover the HBA world wide names. I couldn't find that info in /proc or /sys, where most docs claimed they would be. What I did find in /sys seemed to indicate that it's set by software. HBAnyware seemed to be the easiest route to take.
4. Install Emulex HBAnyware support software (installs to /usr/sbin/hbanyware by default) using the install script in the tar distribution (it's huge at 200+ megs...)
5. (NOTE: This is not really necessary since RAID level "multipath" takes care of my issue nicely. I didn't know this when I posted so I'm making this note to let others know that step five can be skipped altogether if you don't need multipath at the LVM level.) Install multipath support software using 'yum' and then configure:

Code:

    yum install device-mapper-multipath

Edit the /etc/multipath.conf file to contain these changes:

Code:

     # Blacklist all devices by default. Remove this to enable multipathing
      # on the default devices. 
      #blacklist {
      #        devnode "*"
      #}

      defaults {
              user_friendly_names yes
      }
      defaults {
              udev_dir                /dev
              polling_interval        10
              selector                "round-robin 0"
              path_grouping_policy    failover
              getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
              prio_callout            /bin/true
              path_checker            readsector0
              rr_min_io               100
              rr_weight               priorities
              failback                immediate
              no_path_retry           fail
              user_friendly_name      yes
      }

Code:

    'modprobe dm-multipath' (load the dm-multipath module)
    'modprobe dm-round-robin' (load the dm-round-robin module)
    'service multipathd start' (start the multipathd service)
    'multipath -v3 -ll' (check the current multipath device info)
    'chkconfig multipathd on' (set multipathd to start on boot)

6. Partition /dev/sda through /dev/sdd to be partition type 'fd' (Linux RAID autodetect):

Code:

    'fdisk /dev/sda' (Create partition 1 and set it to type 'fd' then save and exit)
    'sfdisk -d /dev/sda | sfdisk /dev/sdb'
    'sfdisk -d /dev/sda | sfdisk /dev/sdc'
    'sfdisk -d /dev/sda | sfdisk /dev/sdd'

7. Set up multipath RAID for redundancy. The RAID "level" called multipath isn't really RAID. It's just using the RAID subsystem to present the four paths (represented by /dev/sda through /dev/sdd in my case) as one device: /dev/md0. Make sure you have the md module loaded as well or 'mdadm' will complain. Also, make sure that you HAVEN'T used the 'dmsetup' command to assign the disk device nodes to dmmp (dm multipath). That interferes with RAID being able to access them. Just in case you did... you will need to do this first (an example for my case. 'mpath0' might not be in your configuration):

Code:

    dmsetup -C remove mpath0

Here is the set up for multipath RAID:

Code:

    'mdadm -C /dev/md0 --level=multipath --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1'
    'mdadm --detail --scan > /etc/mdadm.conf' (saves the above configuration to /etc/mdadm.conf so the RAID will start automatically on boot)

Other UNIX.COM Threads You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
How to add new paths to $PATH	wrapster	SUN Solaris	2	4 Weeks Ago 01:46 AM
Enabling a script to run in multiple paths	dowell	Shell Programming and Scripting	5	03-03-2008 10:04 PM
create paths	alias47	UNIX for Dummies Questions & Answers	4	08-07-2007 06:14 AM
Paths update	phpote	UNIX for Dummies Questions & Answers	5	07-25-2001 04:57 AM
paths	caiohn	UNIX for Dummies Questions & Answers	1	01-17-2001 03:35 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode