I'm setting up a CentOS 5 server that will be connected to an HP EVA4000. My previous experience is with an HP VA7400 and HP-UX. In HP-UX I can add "alternate paths" to a volume group in order to have redundant paths to the SAN via dual fiber channel HBAs just by either adding them with 'vgextend' or appending the appropriate PV paths when using 'vgcreate'. I tried doing something similar in CentOS with 'vgextend' and got some errors about duplicate PV ids. The LUN (a single one) I create on the EVA4000 show up on the CentOS box as /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd. And when I partitioned /dev/sda so that I had /dev/sda1, I also saw that /dev/sdb1, /dev/sdc1 and /dev/sdd1 were automagically there. So I know that these extra devices reflect the dual fiber channel HBAs and the dual paths through our redundant fiber switches.
My question is... did I do this the right way? When I do a 'vgdisplay' now I get:
Code:
[root@localhost ~]# vgdisplay
Found duplicate PV Up921bgzWK4I7xtw7JiOB3VrdbVZhH2Z: using /dev/disk/by-path/pci-0000:13:00.0-fc-0x50001fe1500ca568:0x0001000000000000-part1 not /dev/sda1
Found duplicate PV Up921bgzWK4I7xtw7JiOB3VrdbVZhH2Z: using /dev/disk/by-id/scsi-3600508b400070ac40000900000090000-part1 not /dev/disk/by-path/pci-0000:13:00.0-fc-0x50001fe1500ca568:0x0001000000000000-part1
at the beginning of the expected output. I want to avoid having to test the redundancy by disconnecting the fiber if possible. And in all honesty I don't think this is the right way to set up redundant paths on CentOS (or any other Linux distro for that matter). I think the "alternate paths" functionality is specific to HP-UX's implementation of LVM. But I suspect there is a "right way" to do alternate paths on Linux for redundancy.
Last edited by deckard; 08-10-2007 at 11:16 AM..
Reason: Forgot code tags...
...Running the controllers in active/active mode we're using RedHats Device Mapper Multipath software to create multiple paths to our SAN... I hope this helps you out a little...
It sure did! Since CentOS uses the 'yum' system some searching on the web and within the yum respositories led me to this:
Code:
[root@localhost etc]# yum install device-mapper-multipath
Loading "installonlyn" plugin
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for device-mapper-multipath to pack into transaction set.
device-mapper-multipath-0 100% |=========================| 9.5 kB 00:00
---> Package device-mapper-multipath.i386 0:0.4.7-8.el5 set to be updated
--> Running transaction check
Dependencies Resolved
=============================================================================
Package Arch Version Repository Size
=============================================================================
Installing:
device-mapper-multipath i386 0.4.7-8.el5 base 760 k
Transaction Summary
=============================================================================
Install 1 Package(s)
Update 0 Package(s)
Remove 0 Package(s)
Total download size: 760 k
Is this ok [y/N]: y
Downloading Packages:
(1/1): device-mapper-mult 100% |=========================| 760 kB 00:01
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing: device-mapper-multipath ######################### [1/1]
Installed: device-mapper-multipath.i386 0:0.4.7-8.el5
Complete!
Now I just need to find a good HOWTO and give this a shot. Thanks a LOT!
Here's what I've come up with from start to finish (OS install through finally setting up multipath using the RAID subsystem) and I will be testing it from the ground up today. I mistakenly installed the 32-bit version of CentOS during my testing phase, so I am starting over with 64-bit today. If I run into any snags, I'll probably post back here. This is the procedure:
1. Install CentOS 5 (64-bit) with Xen Kernel
2. Install 'screen' (My personal preference for detached terminal sessions. For those who don't know, think of screen as "VNC" for the command line in terms of coming back to a terminal that you detached from earlier.), 'gcc', 'kernel-xen-devel-2.6.18-8.el5' using 'yum':
3. Install Emulex 'lpfc' driver to support the fiber channel cards. There is a simple install script in the driver tarball that woks if you have a compiler and kernel sources for your running kernel. The driver that comes with the OS doesn't appear to support the HBAnyware software mentioned in the next item which was the only way I was able to discover the HBA world wide names. I couldn't find that info in /proc or /sys, where most docs claimed they would be. What I did find in /sys seemed to indicate that it's set by software. HBAnyware seemed to be the easiest route to take.
4. Install Emulex HBAnyware support software (installs to /usr/sbin/hbanyware by default) using the install script in the tar distribution (it's huge at 200+ megs...)
5. (NOTE: This is not really necessary since RAID level "multipath" takes care of my issue nicely. I didn't know this when I posted so I'm making this note to let others know that step five can be skipped altogether if you don't need multipath at the LVM level.) Install multipath support software using 'yum' and then configure:
Code:
yum install device-mapper-multipath
Edit the /etc/multipath.conf file to contain these changes:
Code:
# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
#blacklist {
# devnode "*"
#}
defaults {
user_friendly_names yes
}
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy failover
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout /bin/true
path_checker readsector0
rr_min_io 100
rr_weight priorities
failback immediate
no_path_retry fail
user_friendly_name yes
}
Code:
'modprobe dm-multipath' (load the dm-multipath module)
'modprobe dm-round-robin' (load the dm-round-robin module)
'service multipathd start' (start the multipathd service)
'multipath -v3 -ll' (check the current multipath device info)
'chkconfig multipathd on' (set multipathd to start on boot)
6. Partition /dev/sda through /dev/sdd to be partition type 'fd' (Linux RAID autodetect):
Code:
'fdisk /dev/sda' (Create partition 1 and set it to type 'fd' then save and exit)
'sfdisk -d /dev/sda | sfdisk /dev/sdb'
'sfdisk -d /dev/sda | sfdisk /dev/sdc'
'sfdisk -d /dev/sda | sfdisk /dev/sdd'
7. Set up multipath RAID for redundancy. The RAID "level" called multipath isn't really RAID. It's just using the RAID subsystem to present the four paths (represented by /dev/sda through /dev/sdd in my case) as one device: /dev/md0. Make sure you have the md module loaded as well or 'mdadm' will complain. Also, make sure that you HAVEN'T used the 'dmsetup' command to assign the disk device nodes to dmmp (dm multipath). That interferes with RAID being able to access them. Just in case you did... you will need to do this first (an example for my case. 'mpath0' might not be in your configuration):
Code:
dmsetup -C remove mpath0
Here is the set up for multipath RAID:
Code:
'mdadm -C /dev/md0 --level=multipath --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1'
'mdadm --detail --scan > /etc/mdadm.conf' (saves the above configuration to /etc/mdadm.conf so the RAID will start automatically on boot)
if I have these wildcards to download from:
path1/*.txt
path2/*.txt
path3/*.txt
path4/*.txt
path5/*.txt
under a link such as this:
http://abc .com/]abc.com
Can wget be written in such a way to extract only those files and create the corresponding paths under a target folder? So the... (4 Replies)
HACMP two-node cluster with mirrored LVM.
HACMP two-node cluster with two SAN storages mirrored using LVM. Configured 2 disk heartbeat networks - 1 per each SAN storage. While performing redundancy tests. Once one of SAN storage is down - cluster is going to ERROR state. What are the guidelines... (2 Replies)
I am working on a script for Mac OS X that, among many other things, gets a list of all the installed Applications. I am pulling the list from the system_profiler command and formatting it using grep and awk. The problem is that I want to be able to use each result individually later in the script.... (3 Replies)
Hi ,
I am a newbie!!!
I want to develop a script for deleting files older than x days from multiple paths. Now I could reach upto this piece of code which deletes files older than x days from a particular path. How do I enhance it to have an input from a .txt file or a .dat file? For eg:... (12 Replies)
I need to update about 2400 files in a directory subtree, with a new directory path inside the files
I need to change this occurence in all files: /d2/R12AB/VIS/apps/tech_st/10.1.2
with this: /u01/PROD/apps/apps_st/10.1.3
I know how to change single words using "find . -type f -print0 |... (6 Replies)
Hello All,
We have a script which when run on the host gives us the status of the internal and SAN attached disks, its paths and other information. The output of which is as follows:
Nm DEVICE VEND PROD LunID SYM DevID DISKGROUP SIZE N E... (3 Replies)
Hey everyone.
I am working on designing a logging solution for a deployment we have going out in a few months. Right now we have a single storage head end, connected via fibre to a SAN. Basically the plan is to create a number of smaller LUNs on the SAN, and then use LVM2 to handle concatenating... (5 Replies)
Hi all
Right, Im a little bit thick when it comes to SAN :-) IVe got a dual port qlc in a solaris 10 server. Its all connected and I can see a number of disks. The SAN guy here has asked me about preferred paths, i.e. as the dual ports are connected to two fabrics, etc device will have 4... (2 Replies)
How do i find files in more than one directory? I searched through forums, but could not land into the right thread.
I tried something like
find dir1|dir2 -name file1
but it doesn't work. Please suggest. (5 Replies)
I have a script that i need to run from different paths.
for example
mypc/path1/path2/,
mypc/path1/path2/path3/, and
mypc/path1/path2/path3/path4
How do i set up that script so that it can execute in any
of the above paths or how can i make it run on any path on my computer?? (5 Replies)
08-09-2007
It sure did! Since CentOS uses the 'yum' system some searching on the web and within the yum respositories led me to this:
SAN / LVM / Multiple Paths