Linux® RDAC Multipath Drivers

這篇文章適用於 IBM Storage DS3000, DS4000 以及 DS5000 (感謝我同事 David 提供的文章)

在 Linux 底下有一個服務為 multipathd ,開啟後就可以讓 multipath 的功能啟動. 設定方式請參考 https://benjr.tw/159
不過這 multipath 的裝置非原始的 sdx 而是變成 /dev/mapper/mpathxx 裝置.雖然裝置就如同一般的硬碟裝置,並且具備 MPIO (multipath I/O: 主要是為了確保連線不會因為單一的網路斷線而失去連線) 的功能.但感覺上???

如果你的儲存裝置為 IBM System Storage DS3000, DS4000 或是 DS5000 就可以使用 IBM 所提供的 multipath 驅動程式 RDAC (Redundant Disk Array Controller),這時就不需要透過 RHEL 預設的 Multipath 功能,但不是所有的 HBA 都支援 RDAC.

最簡單的 Multipath I/O 連接方式如下,不管是 Server 端還是 Storage 端都一定是 Redundant (備源) 的接法,這是最基礎的接法,真實應用可能會更為複雜.


目前測試環境為

• Server + NIC Port x2
• OS : RHEL 6.3
• Storage : IBM DS3524 (iSCSI Storage) 

不過不是所有的 HBA 都支援 RDAC,在 RDAC 的 Readme.txt 支援列表如下
FCoE CNA

• PCI-E :QLE8042
• PCI-E :LP21000, LP21002
• PCI-E :BR-1020

iSCSI Qlogic

• PCI-X :Qlogic 4050C, Qlogic 4052C
• PCI-E :Qlogic 4060C, Qlogic 4062C

Fibre Channel Qlogic

• PCI-E : QLE2560, QLE2562 (8Gb)
• PCI-E : QLE2460, QLE2462 (4Gb)
• PCI-E : QLE236x (2Gb)
• PCI-X : QLA2460, QLA2462 (4Gb)
• PCI-X : QLA2310F, QLA2340, QLA2342 (2Gb)
• PCI-X : QLA2000 (1Gb)

SAS LSI Logic

• PCI-X : LSISAS3800-X, LSI SAS3442X-R, LSI SAS3442E-R
• PCI-X : 3801X (SAS1 3G)
• PCI-E : 3801E (SAS1 3G)
• PCI-E : 92XX-XXX (SAS2 6G)

Fibre Channel LSI Logic:

• PCI-X : LSI7204XP-LC, LSI7404XP-LC, LSI7104XP, (4Gb)
• PCI-X : LSI7202XP, LSI7402XP (2Gb)
• LSI7104EP, LSI7204EP
• PCI and PCI-X: LSIFC929X (2Gb)

Fibre Channel IBM Branded Emulex

• PCI-X : IBM FC 5758, 5759 (4Gb)
• PCI-X : IBM FC 5716, (2Gb)
• PCI : IBM FC 6228 (1Gb) , 6239 (2Gb)

Fibre Channel Emulex

• PCI-E : LPe12000, LPe12002 (8Gb)
• PCI-E : LPe11000, LPe11002, LPe1150 (4Gb)
• PCI-E : LP10000ExDC (2Gb)
• PCI-X : LP11000, LP11000DC, LP11002, LP1150 (4Gb)
• PCI-X : LP10000, LP10000DC, LP1050, LP9802, LP9802DC, LP982 (2Gb)
• PCI : LP9002DC, LP952L, LP9002L, LP9000 (2Gb)

Fibre Channel Brocade

• PCI-E : Brocade 815, Brocade 825 (8Gb)

Fibre Channel ATTO

• PCI-E : Celerity FC-84EN, Celerity FC-82EN, Celerity FC-81EN (8Gb)

Infiniband HCA Mellanox

• PCI-E : InfiniHost III Lx MHGS18-XTC, InfiniHost Ex MHGA28-XTC, ConnectX MHGH28-XTC, ConnectX MHQH29

Infiniband HCA Voltaire

• PCI-E : HCA 400EX-D (MHGA28-XTC), HCA 410EX-D (MHGS18-XTC), HCA 500EX-D (MHGH28-XTC), 600Ex2

Infiniband HCA QLogic

• PCI-E : 7104-HCA-128LPX-DDR (MHGA28-1TC), 7104-HCA-LPX1P-DDR (MHGS18-XTC), 7104-HCA-LPX2P-DDR (MHGA28-XTC)

步驟:
接 2 條網路線到 DS3524的 iscsi port,並先設定好DS3524, 設定方式雷同 IBM DS3400 主要是把 Lun 指派給 iscsi initiator

1. LinuxR RDAC Multipath Drivers 驅動程式安裝方式很簡單, 因為需要 make 所以要安裝 kernel-devel kernel-header make,接著
   下載 RDAC,解開 source packages,並手動建立驅動程式,使用新建立好的 initrd 更改 menu.lst 選單,重新開機.

   [root@benjr Desktop]# yum install kernel-devel kernel-header
   [root@benjr Desktop]# tar zxvf rdac-LINUX-09.03.0C05.0642-source.tar.gz
   [root@benjr Desktop]# cd Linuxrdac-09.03.0C05.0642
   [root@benjr Desktop]# make clean
   [root@benjr Desktop]# make
   [root@benjr Desktop]# make install
   

2. 在 GRUB 選單中,從新建立的選項開機.
3. IP 設完後,用 #iscsiadm command,將建立2個 iscsi 的連線,使用 #fdisk -l 只會看到1個 Lun.
4. 建立第一條connection

   [root@benjr Desktop]# iscsiadm -m node -T iqn.xxxxx -p  192.168.132.101 -l
   

5. 建立第二條 connection.

   [root@benjr Desktop]# iscsiadm -m node -T iqn.xxxxx -p  192.168.133.101 -l
   

6. 看到 Lun 之後,把它格式化+掛載,試著存取它.
7. 移除其中一條網路線,會停頓一下,之後就正常.如果你拔除網路線之後,跳出 Input/Output error 的訊息,代表失敗.
8. 系統的一些訊息可證明 MPIO 有功能的.

   [root@benjr Desktop]# dmesg
   igb: eth0 NIC Link is Down
   connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4303063833, last ping 4303068833, now 4303073833
   connection1:0: detected conn error (1011)
   session1: session recovery timed out after 144 secs
   

9. 一邊存取同時拔線會頓一下,需要約 2分多鐘 的時間才會走另一條路徑

   94 [RAIDarray.mpp]DS3524:0:0:0 Selection Retry count exhausted
   7 [RAIDarray.mpp]DS3524:0:0 Path Failed
   495 [RAIDarray.mpp]DS3524:0:0:0 Cmnd failed-retry on a new path. vcmnd SN 1228 pdev H7:C0:T0:L0 0×00/0×00/0×00 0x000f0000 mpp_status:6
   

   我們還可以觀察一下在 /proc/mpp/DS3524 下

   [root@benjr DS3524]# cat virtualLun0
   Linux MPP driver. Version:09.03.0C05.0638 Build:Tue Apr 17 15:31:54 CDT 2012
   Lun WWN:60080e50001b70880000073e500df5a0
   Virtual Scsi Address: host_no:8 channel:0 target:0 Lun:0
   Queue Depth = 30
   I/O Statistics:
   Number of IOs:489
   Longest trip of all I/Os:148
   Shortest trip of all I/Os:0
   Number of occurrences of path failover events:2
   Number of occurrences of controller failover events:0
   The longest Controller Failover Time:0
   The shortest Controller Failover Time:0
   total size:491
   Linux MPP driver. Version:09.03.0C05.0638 Build:Tue Apr 17 15:31:54 CDT 2012
   Lun WWN:60080e50001b70880000073e500df5a0
   Physical HBA driver: iscsi_tcp
   Device Scsi Address: host_no:9 channel:0 target:0 Lun:0
   Queue Depth = 32
   I/O Statistics:
    Number of IOs:101
    Longest trip of all I/Os:0
    Shortest trip of all I/Os:0
    Number of occurences of IO failed events:18
   Device state: [6] OPTIMAL
   Device state: [7] FAILED
   Device state: [8] FAILED_NEED_CHECK
   Device state: [9] FAILED_CHECKING
   Device state: [0] FAILED
   Device state: [1] FAILED_NEED_CHECK
   Device state: [2] FAILED_CHECKING
   Device state: [3] FAILED
   Device state: [4] FAILED_NEED_CHECK
   Device state: [5] FAILED_CHECKING
   Path state:[0] OPTIMAL
   Path state:[1] OPTIMAL
   Path state:[2] OPTIMAL_NEED_CHECK
   Path state:[3] OPTIMAL_CHECKING
   Path state:[4] OPTIMAL
   Path state:[5] OPTIMAL_NEED_CHECK
   Path state:[6] OPTIMAL_CHECKING
   Path state:[7] OPTIMAL
   Path state:[8] OPTIMAL_NEED_CHECK
   Path state:[9] OPTIMAL_CHECKING
   Controller Failed? 0
   Outstanding IOs on this device:
   total size:1121
   

   把線路插回去會看到 Recovery 的訊息

   Jul 27 17:01:58 localhost avahi-daemon[2652]: Registering new address record for 192.168.132.102 on eth0.IPv4.
   Jul 27 17:01:58 localhost iscsid: connect to 192.168.132.101:3260 failed (Connection refused)
   Jul 27 17:02:02 localhost iscsid: connection1:0 is operational after recovery (1358 attempts)