Linux SAN Multipathing
There are a lot of SAN multipathing solutions on Linux at the moment. Two of them are discussesed in this blog. The first one is device mapper multipathing that is a failover and load balancing solution with a lot of configuration options. The second one (mdadm multipathing) is just a failover solution with manuel re-anable of a failed path. The advantage of mdadm multiphating is that it is very easy to configure.
Before using a multipathing solution for a production environment on Linux it is also important to determine if the used solution is supportet with the used Hardware. For example HP doesn’t support the Device Mapper Multipathing solution on their servers yet.
Device Mapper Multipathing
Procedure for configuring the system with DM-Multipath:
- Install device-mapper-multipath rpm
- Edit the multipath.conf configuration file:
- comment out the default blacklist
- change any of the existing defaults as needed
- Start the multipath daemons
- Create the multipath device with the multipath
Install Device Mapper Multipath
# rpm -ivh device-mapper-multipath-0.4.7-8.el5.i386.rpm warning: device-mapper-multipath-0.4.7-8.el5.i386.rpm: Header V3 DSA signature: Preparing... ########################################### [100%] 1:device-mapper-multipath########################################### [100%]
Initial Configuration
Set user_friendly_name. The devices will be created as /dev/mapper/mpath[n]. Uncomment the blacklist.
# vim /etc/multipath.conf #blacklist { # devnode "*" #} defaults { user_friendly_names yes path_grouping_policy multibus }
Load the needed modul and the startup service.
# modprobe dm-multipath # /etc/init.d/multipathd start # chkconfig multipathd on
Print out the multipathed device.
# multipath -v2 or # multipath -v3
Configuration
Configure device type in config file.
# cat /sys/block/sda/device/vendor HP # cat /sys/block/sda/device/model HSV200 # vim /etc/multipath.conf devices { device { vendor "HP" product "HSV200" path_grouping_policy multibus no_path_retry "5" } }
Configure multipath device in config file.
# cat /var/lib/multipath/bindings # Format: # alias wwid # mpath0 3600508b400070aac0000900000080000 # vim /etc/multipath.conf multipaths { multipath { wwid 3600508b400070aac0000900000080000 alias mpath0 path_grouping_policy multibus path_checker readsector0 path_selector "round-robin 0" failback "5" rr_weight priorities no_path_retry "5" } }
Set not mutipathed devices on the blacklist. (f.e. local Raid-Devices, Volume Groups)
# vim /etc/multipath.conf devnode_blacklist { devnode "^cciss!c[0-9]d[0-9]*" devnode "^vg*" }
Show Configured Multipaths.
# dmsetup ls --target=multipath mpath0 (253, 1) # multipath -ll mpath0 (3600508b400070aac0000900000080000) dm-1 HP,HSV200 [size=10G][features=1 queue_if_no_path][hwhandler=0] \_ round-robin 0 [prio=4][active] \_ 0:0:0:1 sda 8:0 [active][ready] \_ 0:0:1:1 sdb 8:16 [active][ready] \_ 1:0:0:1 sdc 8:32 [active][ready] \_ 1:0:1:1 sdd 8:48 [active][ready]
Format and mount Device
Fdisk cannot be used with /dev/mapper/[dev_name] devices. Use fdisk on the underlying disks and execute the following command when device-mapper multipath maps the device to create a /dev/mapper/mpath[n] device for the partition.
# fdisk /dev/sda # kpartx -a /dev/mapper/mpath0 # ls /dev/mapper/* mpath0 mpath0p1 # mkfs.ext3 /dev/mapper/mpath0p1 # mount /dev/mapper/mpath0p1 /mnt/san
After that /dev/mapper/mpath0p1 is the first partition on the multipathed device.
Multipathing with mdadm on Linux
The md multipathing solution is only a failover solution what means that only one path is used at one time and no load balancing is made.
Start the MD Multipathing Service
# chkconfig mdmpd on # /etc/init.d/mdmpd start
On the first Node (if it is a shared device)
Make Label on Disk
# fdisk /dev/sda Disk /dev/sdt: 42.9 GB, 42949672960 bytes 64 heads, 32 sectors/track, 40960 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System /dev/sdt1 1 40960 41943024 fd Linux raid autodetect # partprobe
Bind multiple paths together
# mdadm --create /dev/md4 --level=multipath --raid-devices=4 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1
Get UUID
# mdadm --detail /dev/md4 UUID : b13031b5:64c5868f:1e68b273:cb36724e
Set md configuration in config file
# vim /etc/mdadm.conf # Multiple Paths to RAC SAN DEVICE /dev/sd[qrst]1 ARRAY /dev/md4 uuid=b13031b5:64c5868f:1e68b273:cb36724e # cat /proc/mdstat
On the second Node (Copy the /etc/mdadm.conf from the first node)
# mdadm -As # cat /proc/mdstat
Restore a failed path
# mdadm /dev/md1 -f /dev/sdt1 -r /dev/sdt1 -a /dev/sdt1
26 Comments »
Leave a comment
-
Recent
- Linux – Repair Bootloader / Change Boot device path
- Join RedHat Linux to Microsoft Active Directory
- HP-UX Increase Veritas cluster filesystem (CFS) online
- MC/Serviceguard Cluster – Replace Quorum Server
- HP-UX Integrity Virtual Machines (Integrity VM)
- MC/Serviceguard Cluster on HP-UX 11.31
- HP-UX 11i comfortable shell environment
- Xen Guest (DomU) Installation
- Linux SAN Multipathing (HP Storage)
- Linux Network Bonding
- Linux SAN Multipathing
- ASM Disk not shown in Oracle Universal Installer (OUI) or DBCA
-
Links
-
Archives
- December 2011 (1)
- July 2010 (1)
- April 2010 (1)
- August 2009 (1)
- October 2008 (1)
- August 2008 (1)
- May 2008 (1)
- March 2008 (1)
- February 2008 (1)
- December 2007 (1)
- November 2007 (5)
-
Categories
-
RSS
Entries RSS
Comments RSS
Hi Thomas, thanks for posting this article. I was looking for some way of testing multipathing on suse linux or redhat and Looks like dm-multipathing might do it. Thanks
Comment by gaurav | May 23, 2008 |
Excellent Write up… Very helpful.
Helped with my configuration.
Thank you for posting.
Comment by Retheesh Kumar | January 6, 2009 |
Hello Thomas,
Many thanks for this article. It has just got my disks from an HP SAN configured nice and quickly.
Interestingly, Red Hat’s own DM Multipathing document doesn’t cover some of the configuration steps you mention (e.g. use of kpartx, the /sys/block/sda/device/* files and the /var/lib/multipath/bindings file). Knowing where to look for this information saved me some valuable time.
Thanks again,
Colin
Comment by Colin Brett | September 8, 2009 |
Hi Thomas,
Thank you very much for this brilliant article. Its very helpful.
thanks
zuraidi
Comment by Zuraidi | November 19, 2009 |
[…] あ、このへんに書いてあったね。 Linux SAN Multipathing ; Thomas Vogt ;s IT Blog https://thomasvogt.wordpress.com/2007/11/29/linux-san-multipathing/ […]
Pingback by dm-multipathで後からLUNを追加する « bukochan.com | July 19, 2010 |
I’m using RHEL 5.5 and fdisk works fine with /dev/mapper devices.
Comment by Brian Schonecker | September 10, 2010 |
[…] Answers Anonymous Install RHEL with “linux mpath” option. Otherwise, you can follow the following: https://thomasvogt.wordpress.com/2007/11/29/linux-san-multipathing/ […]
Pingback by Adding multipath to server and change lvm to use new devices | November 26, 2010 |
Hi,
just a question: you are using ext3 as fs. is that not very risky? afaik it is not a fs recommended for nas systems (multi mount problems etc)
Comment by sascha | January 27, 2011 |
Looks like you’ve got a couple of sites who have copied your content: http://landofthefreeish.com/linux/howto-linux-san-multipathing/ and http://lazysystemadmin.blogspot.com/2010/10/linux-san-multipathing-using-device.html
Comment by Sonia Hamilton | April 21, 2011 |
Hi Thomas,
Great article.
I’m using an EMC Clariion storage with Linux clusters, and tried to use EMC solution ( SolutionEnabler ) to map files/devices to storage devices.
I found that the SolutionEnabler mapping/resolving utility doesn’t work on multi-path devices while having the user_friendly_names set to yes, and wonder if you can elaborate on the difference between setting the user_friendly_names to YES and setting it to NO, and what is the impact on the OS.
Thank you
Roye
Comment by Roye AVidor | January 21, 2013 |
[…] proceed as normal with the […]
Pingback by Linux multipathd: how to specify multipath device owner and group in /etc/multipathd.conf | frozen hamster | April 11, 2013 |
Hello Thomas,
Excellent Article,Thanks for posting such a nice Article
Comment by Rajesh | April 14, 2013 |
Diseases that cause hormonel imbalances and improper habitat
temperatures also increase the risk of MBD. Succeeding feeding all through out the day can also be done but only in small quantity.
There are innumerable dog carrier varieties available on
the internet.
Comment by www.founddelight.com | May 3, 2013 |
Super post! Thanks a million
Comment by Wambua | August 15, 2013 |
Thanks a lot for your post!
Comment by bagya | August 21, 2013 |
[…] Install RHEL with “linux mpath” option. Otherwise, you can follow the following: https://thomasvogt.wordpress.com/2007/11/29/linux-san-multipathing/ […]
Pingback by Adding multipath to server and change lvm to use new devices - Just just easy answers | September 7, 2013 |
Hi Thomas,
Nice posting. Just wanted to mention to you that I saw a very similar post at http://www.lazysystemadmin.com/2010/10/linux-san-multipathing-using-device.html. Check it out yourself if you have a moment.
Jonathan
Comment by Jonathan | October 1, 2013 |
Good one thanks
Comment by Ganga | May 29, 2014 |
Nice Post. Thank you so much, it was of great help
Comment by RK | September 12, 2014 |
Very good information. Can you please give the information about the partition. It should be primary, extended or logical. Also please give the partition type. Is it GPT? Thank you.
Comment by Devakumar | June 23, 2015 |
how to add additional storage in an existing setup. can you please provide all steps.Thanks
Comment by ram | May 8, 2016 |
[…] Thomas Vogt’s IT Blog — Linux SAN Multipathing […]
Pingback by Настраиваем Device Mapper Multipathing в CentOS Linux 7.2 для FC HBA при подключении к СХД HP 3PAR 7200 (3PAR OS v. 3.2.2) | Блог IT-KB | June 12, 2016 |
http://dhjsdhv2667226ll.com
Comment by BizUttextDies | June 2, 2017 |
Excellent Write up… Very helpful. I used in production in one step.
Comment by Sunil Kumar | June 13, 2017 |
[…] Multipathing reference: Linux SAN Multipathing […]
Pingback by How to Configure SAN disks and Multipathing in Linux – Small Systems Solutions | August 7, 2018 |
пригожий ресурс http://zarabotat-v-internete.biz/645-partnerka-bitkoin-kazino-bitstarz.html
Comment by JamesDuh | April 6, 2020 |