Here's what I've come up with from start to finish (OS install through finally setting up multipath using the RAID subsystem) and I will be testing it from the ground up today. I mistakenly installed the 32-bit version of CentOS during my testing phase, so I am starting over with 64-bit today. If I run into any snags, I'll probably post back here. This is the procedure:
1. Install CentOS 5 (64-bit) with Xen Kernel
2. Install 'screen' (My personal preference for detached terminal sessions. For those who don't know, think of screen as "VNC" for the command line in terms of coming back to a terminal that you detached from earlier.), 'gcc', 'kernel-xen-devel-2.6.18-8.el5' using 'yum':
Code:
yum install screen
yum install gcc
yum install kernel-xen-devel-2.6.18-8.el5
3. Install
Emulex 'lpfc' driver to support the fiber channel cards. There is a simple install script in the driver tarball that woks if you have a compiler and kernel sources for your running kernel. The driver that comes with the OS doesn't appear to support the HBAnyware software mentioned in the next item which was the only way I was able to discover the HBA world wide names. I couldn't find that info in /proc or /sys, where most docs claimed they would be. What I did find in /sys seemed to indicate that it's set by software. HBAnyware seemed to be the easiest route to take.
4. Install Emulex HBAnyware support software (installs to /usr/sbin/hbanyware by default) using the install script in the tar distribution (it's huge at 200+ megs...)
5. (NOTE: This is not really necessary since RAID level "multipath" takes care of my issue nicely. I didn't know this when I posted so I'm making this note to let others know that step five can be skipped altogether if you don't need multipath at the LVM level.) Install multipath support software using 'yum' and then configure:
Code:
yum install device-mapper-multipath
Edit the /etc/multipath.conf file to contain these changes:
Code:
# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
#blacklist {
# devnode "*"
#}
defaults {
user_friendly_names yes
}
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy failover
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout /bin/true
path_checker readsector0
rr_min_io 100
rr_weight priorities
failback immediate
no_path_retry fail
user_friendly_name yes
}
Code:
'modprobe dm-multipath' (load the dm-multipath module)
'modprobe dm-round-robin' (load the dm-round-robin module)
'service multipathd start' (start the multipathd service)
'multipath -v3 -ll' (check the current multipath device info)
'chkconfig multipathd on' (set multipathd to start on boot)
6. Partition /dev/sda through /dev/sdd to be partition type 'fd' (Linux RAID autodetect):
Code:
'fdisk /dev/sda' (Create partition 1 and set it to type 'fd' then save and exit)
'sfdisk -d /dev/sda | sfdisk /dev/sdb'
'sfdisk -d /dev/sda | sfdisk /dev/sdc'
'sfdisk -d /dev/sda | sfdisk /dev/sdd'
7. Set up multipath RAID for redundancy. The RAID "level" called multipath isn't really RAID. It's just using the RAID subsystem to present the four paths (represented by /dev/sda through /dev/sdd in my case) as one device: /dev/md0. Make sure you have the md module loaded as well or 'mdadm' will complain. Also, make sure that you HAVEN'T used the 'dmsetup' command to assign the disk device nodes to dmmp (dm multipath). That interferes with RAID being able to access them. Just in case you did... you will need to do this first (an example for my case. 'mpath0' might not be in your configuration):
Code:
dmsetup -C remove mpath0
Here is the set up for multipath RAID:
Code:
'mdadm -C /dev/md0 --level=multipath --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1'
'mdadm --detail --scan > /etc/mdadm.conf' (saves the above configuration to /etc/mdadm.conf so the RAID will start automatically on boot)
