打开Mellanox官网下载驱动 Linux InfiniBand Drivers

点击下载链接跳转至

Tgz解压缩执行 ./mlnxofedinstall发现缺少模块

# ./mlnxofedinstall 
Logs dir: /tmp/MLNX_OFED_LINUX.11337.logs
General log file: /tmp/MLNX_OFED_LINUX.11337.logs/general.log
Verifying KMP rpms compatibility with target kernel...
Error: One or more required packages for installing MLNX_OFED_LINUX are missing.
Please install the missing packages using your Linux distribution Package Management tool.
Run:
yum install kernel-modules-extra tk

先安装 kernel modules

# yum install kernel-modules-extra tk

再安装OFED

# ./mlnxofedinstall 
Logs dir: /tmp/MLNX_OFED_LINUX.33933.logs
General log file: /tmp/MLNX_OFED_LINUX.33933.logs/general.log
Verifying KMP rpms compatibility with target kernel...
This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.

Do you want to continue?[y/N]:y

Uninstalling MLNX_EN driver

rpm --nosignature -e --allmatches --nodeps libibverbs libibverbs-utils libibumad ibacm librdmacm opensm opensm-libs infiniband-diags opensm opensm-libs libibumad ibacm libibverbs libibverbs-utils librdmacm infiniband-diags

Starting MLNX_OFED_LINUX-23.10-1.1.9.0 installation ...

Verifying...                          ########################################
Preparing...                          ########################################
Verifying...                          ########################################
Preparing...                          ########################################
Installing mlnx-ofa_kernel RPM
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
mlnx-ofa_kernel-23.10-OFED.23.10.1.1.9########################################
Configured /etc/security/limits.conf
Installing kmod-mlnx-ofa_kernel 23.10 RPM
Verifying...                          ########################################
Preparing...                          ########################################
Installing mlnx-ofa_kernel-devel RPM
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
mlnx-ofa_kernel-devel-23.10-OFED.23.10########################################
Installing mlnx-ofa_kernel-source RPM
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
mlnx-ofa_kernel-source-23.10-OFED.23.1########################################
Installing kmod-kernel-mft-mlnx 4.26.1 RPM
Verifying...                          ########################################
Preparing...                          ########################################
Installing knem RPM
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
knem-1.1.4.90mlnx3-OFED.23.10.0.2.1.1.########################################
Installing kmod-knem 1.1.4.90mlnx3 RPM
Verifying...                          ########################################
Preparing...                          ########################################
Installing xpmem RPM
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
xpmem-2.7.3-1.2310055.rhel8u1         ########################################
Installing kmod-xpmem 2.7.3 RPM
Verifying...                          ########################################
Preparing...                          ########################################
Installing kmod-iser 23.10 RPM
Verifying...                          ########################################
Preparing...                          ########################################
Installing kmod-srp 23.10 RPM
Verifying...                          ########################################
Preparing...                          ########################################
Installing kmod-isert 23.10 RPM
Verifying...                          ########################################
Preparing...                          ########################################
Installing libxpmem 2.7.3 RPM
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
libxpmem-2.7.3-1.2310055.rhel8u1      ########################################
Installing user level RPMs:
Verifying...                          ########################################
Preparing...                          ########################################
Device (0000:2f:00.0):
    0000:2f:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
    Link Width: x16
    PCI Link Speed: 8GT/s

Device (0000:2f:00.1):
    0000:2f:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
    Link Width: x16
    PCI Link Speed: 8GT/s

Installation finished successfully.

Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:mlnx-fw-updater-23.10-1.1.9.0    ################################# [100%]

Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf

Initializing...
Attempting to perform Firmware update...

The firmware for this device is not distributed inside Mellanox driver: 0000:2f:00.0 (PSID: LNV0000000017)
To obtain firmware for this device, please contact your HW vendor.

Failed to update Firmware.
See /tmp/MLNX_OFED_LINUX.33933.logs/fw_update.log
To load the new driver, run:
/etc/init.d/openibd restart
    
# /etc/init.d/openibd restart
Unloading rdma_cm                                          [FAILED]
rmmod: ERROR: Module rdma_cm is in use by: rpcrdma ib_srpt ib_isert

# modprobe -r rdma_cm
modprobe: FATAL: Module rdma_cm is in use.

# rmmod rdma_cm
rmmod: ERROR: Module rdma_cm is in use by: rpcrdma ib_srpt ib_isert

# cat /tmp/MLNX_OFED_LINUX.33933.logs/fw_update.log

The firmware for this device is not distributed inside Mellanox driver: 0000:2f:00.0 (PSID: LNV0000000017)
To obtain firmware for this device, please contact your HW vendor.

EXIT_STATUS: 2

继续安装OFED，但是kernel不支持，先增加kernel模块，缺少装啥即可

yum install kernel-devel-4.18.0-305.19.1.el8_4.x86_64 kernel-rpm-macros

# bash mlnx_add_kernel_support.sh -m ./
Note: This program will create MLNX_OFED_LINUX TGZ for rhel8.1 under /tmp directory.
Do you want to continue?[y/N]:y
See log file /tmp/mlnx_iso.20469_logs/mlnx_ofed_iso.20469.log

Checking if all needed packages are installed...
/lib/modules/4.18.0-305.19.1.el8_4.x86_64/build//scripts is required to build mlnx-ofa_kernel-modules RPM.
Please install the corresponding kernel-devel RPM.

Error: One or more required packages for installing OFED-internal are missing.
Please install the missing packages using your Linux distribution Package Management tool.
Run:
yum install kernel-devel-4.18.0-305.19.1.el8_4.x86_64 kernel-rpm-macros

再继续安装

./mlnxofedinstall --add-kernel-support

驱动需要重启Server次阿能生效，注意IB的VF路径与RDMA卡是不同的

# echo 6 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs 

VF生成成功了

# lspci -nn|grep Mellanox
0000:2f:00.0 Ethernet controller [0200]: Mellanox Technologies MT28908 Family [ConnectX-6] [15b3:101b]
0000:2f:00.1 Ethernet controller [0200]: Mellanox Technologies MT28908 Family [ConnectX-6] [15b3:101b]
0000:2f:00.2 Ethernet controller [0200]: Mellanox Technologies MT28908 Family [ConnectX-6 Virtual Function] [15b3:101c]
0000:2f:00.3 Ethernet controller [0200]: Mellanox Technologies MT28908 Family [ConnectX-6 Virtual Function] [15b3:101c]
0000:2f:00.4 Ethernet controller [0200]: Mellanox Technologies MT28908 Family [ConnectX-6 Virtual Function] [15b3:101c]
0000:2f:00.5 Ethernet controller [0200]: Mellanox Technologies MT28908 Family [ConnectX-6 Virtual Function] [15b3:101c]
0000:2f:00.6 Ethernet controller [0200]: Mellanox Technologies MT28908 Family [ConnectX-6 Virtual Function] [15b3:101c]
0000:2f:00.7 Ethernet controller [0200]: Mellanox Technologies MT28908 Family [ConnectX-6 Virtual Function] [15b3:101c]

# ip link
link/ether b4:96:91:bf:94:94 brd ff:ff:ff:ff:ff:ff
16: ens1f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether b8:3f:d2:d3:e4:c6 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off
    vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off
    vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off
    vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off
    vf 4 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off
    vf 5 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off