ConnectX®-3 EN 10 / 40 Gigabit Ethernet

手邊有兩張 ConnectX®-3 EN Single/Dual-Port 10 and 40 Gigabit Ethernet Adapters PCI Express 3.0 的卡.詳細資料 http://www.mellanox.com/page/products_dyn?product_family=127

圖片出自於 http://www.mellanox.com/uploads/product_families/cat_71/gfx_00975.jpg

不知道要怎麼用,在 Ubuntu 14.04 下看的到卻抓不到裝置.

root@benjr:~# lspci |grep -i Mellanox
03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

雖然 Ubuntu 14.04 有提供 ConnectX®-3 的模組但沒有效,所以照著 Mellanox 官方網站說明自行編譯模組 ConnectX-3_Ethernet_Single_and_Dual_QSFP+_Port_Adapter_Card_User_Manual.pdf

先用有 administrator 的權限登入,避免每個指令都要 sudo 打密碼的.但也請小心.

root@benjr:~# sudo su -
[sudo] password for ben:

第一步就是要去 Mellanox 下載,最新的版本請自行上 http://www.mellanox.com/page/products_dyn?product_family=27 查詢

root@benjr:~# wget http://www.mellanox.com/downloads/Drivers/mlnx-en-2.2-1.0.1.tgz

下載到的檔案為 tgz 先解開吧!!

root@benjr:~# tar zxvf mlnx-en-2.2-1.0.1.tgz

root@benjr:~# cd mlnx-en-2.2-1.0.1/

透過 Mellanox 提供的安裝程式 install.sh 來安裝.

root@benjr:~/mlnx-en-2.2-1.0.1# ./install.sh
Installing mlnx_en for Linux
Starting installation at a??  5a?? 29 17:27:08 CST 2014...
Verifying dependencies
/usr/bin/dpkg-buildpackage -us -uc   FAILED
Please review /tmp/install-mlx4_en.log.7254

root@benjr:~/mlnx-en-2.2-1.0.1# cat /tmp/install-mlx4_en.log.7254
Starting installation at a??  5a?? 29 17:27:08 CST 2014...
EXEC: /bin/mkdir /tmp/MLNX_EN
EXEC: /bin/mkdir /tmp/MLNX_EN/BUILD
EXEC: /bin/mkdir /tmp/MLNX_EN/SRPMS
EXEC: /bin/mkdir /tmp/MLNX_EN/RPMS
EXEC: /bin/mkdir /tmp/MLNX_EN/SOURCES
EXEC: cp /root/mlnx-en-2.2-1.0.1/SOURCES/mlnx-en_2.2.orig.tar.gz .
EXEC: tar xzf /root/mlnx-en-2.2-1.0.1/SOURCES/mlnx-en_2.2.orig.tar.gz
EXEC: /usr/bin/dpkg-buildpackage -us -uc
./install.sh: line 91: /usr/bin/dpkg-buildpackage: No such file or directory

dpkg??? : 「dpkg 」是「Debian Package」的簡寫
以 deb (Debian) ?副檔名的文件安裝 ( # dpkg -i file.deb ) 與 RPM 使用方式類似,可用安裝(移除…)於 Linux 系統的軟體,不過我們在 Ubuntu 下常用的 APT (#apt-get …) 都是基於 dpkg 所寫成,以提供給遠端安裝軟體或是排除軟體的相關連等問題.

編輯會需要使用到 dpkg-buildpackage 程式,該程式存在 dpkg-dev 套件裡,必須先安裝才能使用.

root@benjr:~/mlnx-en-2.2-1.0.1# apt-get install dpkg-dev

再安裝看看

root@benjr:~/mlnx-en-2.2-1.0.1# ./install.sh
Installing mlnx_en for Linux
Starting installation at a??  5a?? 29 17:44:37 CST 2014...
Verifying dependencies
/usr/bin/dpkg-buildpackage -us -uc   FAILED
Please review /tmp/install-mlx4_en.log.9858

還是失敗了先看看 log 吧!!

root@benjr:~/mlnx-en-2.2-1.0.1# cat /tmp/install-mlx4_en.log.9858
Starting installation at a??  5a?? 29 17:44:37 CST 2014...
EXEC: /bin/rm -rf /tmp/MLNX_EN
EXEC: /bin/mkdir /tmp/MLNX_EN
EXEC: /bin/mkdir /tmp/MLNX_EN/BUILD
EXEC: /bin/mkdir /tmp/MLNX_EN/SRPMS
EXEC: /bin/mkdir /tmp/MLNX_EN/RPMS
EXEC: /bin/mkdir /tmp/MLNX_EN/SOURCES
EXEC: cp /root/mlnx-en-2.2-1.0.1/SOURCES/mlnx-en_2.2.orig.tar.gz .
EXEC: tar xzf /root/mlnx-en-2.2-1.0.1/SOURCES/mlnx-en_2.2.orig.tar.gz
EXEC: /usr/bin/dpkg-buildpackage -us -uc
dpkg-buildpackage: source package mlnx-en
dpkg-buildpackage: source version 2.2-1.0.1.0.gdf6fefb
dpkg-buildpackage: source distribution unstable
dpkg-buildpackage: source changed by Vladimir Sokolovsky <vlad@mellanox.com>
 dpkg-source --before-build mlnx-en-2.2
dpkg-buildpackage: host architecture amd64
dpkg-checkbuilddeps: Unmet build dependencies: debhelper (>= 8.0.0) autotools-dev dkms
dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
dpkg-buildpackage: warning: (Use -d flag to override.)

原來還少了 (Unmet – 未滿足) 上面所說的 debhelper , autotools-dev , dkms 三個套件.

root@benjr:~/mlnx-en-2.2-1.0.1# apt-get install debhelper autotools-dev
root@benjr:~/mlnx-en-2.2-1.0.1# ./install.sh
Installing mlnx_en for Linux
Starting installation at 五  5月 30 14:54:08 CST 2014...
Verifying dependencies
/usr/bin/dpkg-buildpackage -us -uc   FAILED
Please review /tmp/install-mlx4_en.log.4866

雖然說失敗但是查看 log

root@benjr:~/mlnx-en-2.2-1.0.1# cat /tmp/install-mlx4_en.log.4866
dh_auto_configure: ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=

${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --libexecdir=

${prefix}/lib/mstflint --disable-maintainer-mode --disable-dependency-tracking returned exit code 1
make: *** [build] Error 25
dpkg-buildpackage: error: debian/rules build gave error exit status 2

雖然顯示錯誤,但 mlx4_core.ko ,mlx4_ib.ko ,mlx4_en.ko 都已經編譯好可以使用了.

這三個 ko (kernel 2.6 模組) 檔案就是???,再載入模組後可以透過 lsmod 看這三個模組的相依性.

root@benjr:~# lsmod |grep -i mlx
mlx4_ib                12430  0
mlx4_en               115373  0
mlx4_core             268219  1 mlx4_en
compat                 13237  3 mlx4_en,mlx4_ib,mlx4_core
ptp                    18933  3 igb,e1000e,mlx4_en

現在載入 mlx4_en 模組,我們現在只會用到 mlx4_en 至於 mlx4_ib 好像是 IP over IB 才會用到的??

root@benjr:~# cd
root@benjr:~# modprobe mlx4_en
root@benjr:~# dmesg

[21259.496105] mlx4_en: eth6: Close port called
[21260.149596] mlx4_en: eth7: Close port called
[21260.840794] mlx4_en 0000:03:00.0: removed PHC
[21269.746581] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1.0.1 (May 30 2014)
[21269.746857] mlx4_en 0000:03:00.0: registered PHC clock
[21269.747132] mlx4_en 0000:03:00.0: Activating port:1
[21269.747233] mlx4_en: 0000:03:00.0: Port 1: QoS disabled - no HW support
[21269.747303] mlx4_en: 0000:03:00.0: Port 1: QCN disabled - no HW support
[21269.757638] mlx4_en: eth3: Using 128 TX rings
[21269.757644] mlx4_en: eth3: Using 16 RX rings
[21269.757702] mlx4_en: eth3: Initializing port
[21269.757895] mlx4_en 0000:03:00.0: Activating port:2
[21269.757997] mlx4_en: 0000:03:00.0: Port 2: QoS disabled - no HW support
[21269.758000] mlx4_en: 0000:03:00.0: Port 2: QCN disabled - no HW support
[21269.771703] systemd-udevd[8082]: renamed network interface eth3 to eth6
[21269.773112] mlx4_en: eth3: Using 128 TX rings
[21269.773116] mlx4_en: eth3: Using 16 RX rings
[21269.773225] mlx4_en: eth3: Initializing port
[21269.869023] IPv6: ADDRCONF(NETDEV_UP): eth6: link is not ready
[21269.871681] systemd-udevd[8082]: renamed network interface eth3 to eth7
[21269.873007] IPv6: ADDRCONF(NETDEV_UP): eth6: link is not ready
[21269.979354] IPv6: ADDRCONF(NETDEV_UP): eth7: link is not ready
[21269.981002] IPv6: ADDRCONF(NETDEV_UP): eth7: link is not ready

透過 ethtool 可以看到確實是 40G 的 Ethernet

root@benjr:~# ethtool eth7
Settings for eth7:
        Supported ports: [ TP ]
        Supported link modes:   1000baseT/Full
                                1000baseKX/Full
                                10000baseT/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  1000baseT/Full
                                1000baseKX/Full
                                10000baseT/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 40000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes

現在可以把它當作一般網路卡來使用了,不過使用iperf 或是 Qperf 效能似乎都不能達到 40Gbits (5GBytes),可能還要針對效能方面作調整.

iperf

root@benjr:~/qperf-0.4.9# iperf -c 172.16.0.42 -t 60 -w 416K -i 20 -l 64k
------------------------------------------------------------
Client connecting to 172.16.0.42, TCP port 5001
TCP window size:  416 KByte
------------------------------------------------------------
[  3] local 172.16.0.46 port 49758 connected with 172.16.0.42 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec  2.19 GBytes   941 Mbits/sec
[  3] 20.0-40.0 sec  2.19 GBytes   941 Mbits/sec
[  3] 40.0-60.0 sec  2.19 GBytes   941 Mbits/sec
[  3]  0.0-60.0 sec  6.58 GBytes   941 Mbits/sec

Qperf

root@benjr:~/qperf-0.4.9# qperf 192.8.1.2 tcp_bw tcp_lat conf
tcp_bw:
    bw  =  1.79 GB/sec
tcp_lat:
    latency  =  20 us
conf:
    loc_node   =  benjr
    loc_cpu    =  16 Cores: Mixed CPUs
    loc_os     =  Linux 3.13.0-24-generic
    loc_qperf  =  0.4.9
    rem_node   =  benjr
    rem_cpu    =  16 Cores: Intel Xeon  E5520 @ 2.27GHz
    rem_os     =  Linux 3.13.0-24-generic
    rem_qperf  =  0.4.9

1.79 GB/sec 的效能真的不高,檢查一下目前這一張卡是 PCI-E 3.0 x8 的速度,但是我的 Server 卻還是 PCI-E 2.0

root@benjr:~# lspci |grep -i connect
03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

root@benjr:~# lspci -n |grep -i 03:00
03:00.0 0280: 15b3:1003
root@benjr:~# lspci -n -d 15b3:1003 -vvv |grep -i width
                LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

LnkCap :
裝置目前所採用的速度 PCI-Express 3.0 (8G)x8 = 64Gbps (3.0 編碼方式為 128/130,所以資料傳輸頻寬以及原始傳輸率基本當都可以算成 8Gbps) 64Gbps (8 GB/sec).

LnkSta :
目前系統所提供的速度 PCI-Express 2.0 (5G)x8 = 40Gbps (20% overhead-PCIe並且在每八個位元的資料串上用十位元來加以編碼) 32Gbps (4 GB/sec).

關於如何檢查 Server PCI-E 請自行參考 https://benjr.tw/10842

所以剛剛測的值也有點低,會不會是 MTU 太低,改成 9000 試試看.關於 Jumbo frame 請參考 https://benjr.tw/16066

root@benjr:~# qperf 192.8.1.2 tcp_bw tcp_lat conf
tcp_bw:
    bw  =  2.22 GB/sec
tcp_lat:
    latency  =  17.3 us
conf:
    loc_node   =  benjr
    loc_cpu    =  16 Cores: Mixed CPUs
    loc_os     =  Linux 3.13.0-24-generic
    loc_qperf  =  0.4.9
    rem_node   =  benjr
    rem_cpu    =  16 Cores: Intel Xeon  E5520 @ 2.27GHz
    rem_os     =  Linux 3.13.0-24-generic
    rem_qperf  =  0.4.9

和剛剛 1.79 GB/sec 相比較 2.22 GB/sec 已經提升了 24%,但離 40Gbit (5GBytes) 還是有點距離.

是不是要使用 IP over IB 來測試效能會比較好,關於 IPOIB 請參考 https://benjr.tw/28088