1

running RHEL 7.9 on a few servers that have Mellanox HDR infiniband cards installed. I have installed RHEL 7.9 on each of them from dvd, and chose server with gui along with Infiniband Support and a handful of other items (the 5 at the bottom).

So using Application - System Tools - Settings - Network here are pics of that gui.

I get Connected but if I do an ip a it shows ib0 state DOWN. This is on a simple LAN, and in the below pic Proxy = None and IPv6 Settings = ignore and I did not set a gateway value for ipv4 settings but I do not think that should matter.

  • What am I missing or not doing?
  • Am I required to use MLNX_OFED_LINUX-5.5-1.0.3.2-rhel7.9-x86_64.iso and if so what parts of it?
  • I had run the MLNX iso's mlnxofedinstall under a previous RHEL install, and had the ib0 network working but the iso's mlnxofedinstall I think was removing existing rdma stuff and was causing my NFS+RDMA problem so I am a little lost.
  • update: on 1 of my 4 servers I ran mlnxofedinstall --with-nfsrdma and it gets an opensm running and now I have green link lights on all server infiniband's. Is it the opensm that did it? But as a result I somehow lose rdma.service on that server i ran the mlnxofedinstall on and then nukes my changes of getting nfs+rdma working which I really want.

enter image description here enter image description here enter image description here enter image description here

Usage: ./mlnxofedinstall [OPTIONS]

Installation control:
    --force              Force installation
    --tmpdir             Change tmp directory. Default: /tmp
    -k|--kernel <version>
                         Default on this system: 3.10.0-1160.49.1.el7.x86_64
    -s|--kernel-sources <path>
                         Default on this system: /lib/modules/3.10.0-1160.49.1.el7.x86_64/build
    -U|--update          Update installed version
    --with-32bit         Install 32-bit libraries. Relevant for x86_64 platforms
    --without-32bit      Skip 32-bit libraries installation (Default). Relevant for x86_64 platforms
    --distro             Set Distro name for the running OS (e.g: rhel6.5, sles11sp3). Default: Use auto-detection
    --skip-distro-check  Do not check MLNX_OFED_LINUX vs Distro matching
    --without-depcheck   Run the installation without verifying that all required Distro's packages are installed
    --check-deps-only    Check for missing required Distro's packages and exit
    --print-distro       Print distribution name and exit

    --add-kernel-support
                         Add kernel support (Run mlnx_add_kernel_support.sh) and install it
    --add-kernel-support-build-only
                         Add kernel support (Run mlnx_add_kernel_support.sh) and exit

    --kernel-extra-args '<args>'
                         pass <args> to kernel configure script (single paraeter, space separated)
                         - Pass it along with --add-kernel-support

    --kmp                Install kernel RPMs with KMP support if available
                         - Enabled by default with official MLNX_OFED_LINUX releases
                         - Disabled by default if new MLNX_OFED_LINUX was created with mlnx_add_kernel_support.sh
                         - Passing it along with --add-kernel-support will pass --kmp to mlnx_add_kernel_support.sh
    --disable-kmp        Install kernel RPMs without KMP support if available
                         - Enabled by default if new MLNX_OFED_LINUX was created with mlnx_add_kernel_support.sh
    --skip-kmp-verify    Do not check if the KMP rpms are compatiable with the target kernel
    --speed-up-kmp       Speed up KMP RPMs installation by skipping %post section and running the required operations once for all KMPs

    --umad-dev-rw        Grant non root users read/write permission for umad devices instead of default
    --umad-dev-na        Prevent from non root users read/write access for umad devices. Overrides '--umad-dev-rw'
    --enable-mlnx_tune   Enable Running the mlnx_tune utility
    --enable-affinity    Run mlnx_affinity script upon boot
    --disable-affinity   Disable mlnx_affinity script (Default)
    --skip-unsupported-devices-check
                         Don't abort if system has an older, unsupported card
    --enable-opensm      Run opensm upon boot

    --package-install-options
                         RPM install options to use when installing RPM packages (comma separated list)
    --pre-install-<package> <path to script>
                         Run given script before given package's install
    --post-install-<package> <path to script>
                         Run given script after given package's install

Firmware update control:
    --without-fw-update  Skip firmware update
    --fw-update-only     Update firmware. Skip driver installation
    --force-fw-update    Force firmware update
    --fw-image-dir       Firmware images directory to use instead of default package content

Package selection:
    -c|--config <packages config_file>
                         Example of the config file can be found under docs
    --all                Install all available packages
    --bluefield          Install BlueField packages
    --hpc                Install minimum packages required for HPC
    --basic              Install minimum packages for basic functionality
    --dpdk               Install minimum packages required for DPDK
    --ovs-dpdk           Install DPDK and OVS packages
    --vma                Install minimum packages required for VMA
    --xlio                Install minimum packages required for XLIO
    --guest              Install minimum packages required by guest OS
    --hypervisor         Install minimum packages required by hypervisor OS
    --with-fabric-collector
                         Enable installing fabric-collector package
    --with-nvmf          Enable NVMEoF support
    --with-nfsrdma       Enable NFSoRDMA support
    --without-nfsrdma    Disable NFSoRDMA support (Default)

Extra package filtering:
    --kernel-only        Install kernel space packages only
    --user-space-only    Filter selected packages and install only User Space packages
    --without-<package>  Do not install package
    --with-<package>     Force installing package

Miscellaneous:
    -h|--help            Display this help message and exit
    --post-start-delay <sec>
                         Set openibd POST_START_DELAY parameter in seconds. (Default 0)
    -p|--print-available Print available packages for current platform
                     And create corresponding ofed.conf file

1 Answer 1

0

the solution to my problems was choosing Infiniband Support at install time, after reinstalling RHEL 7 from dvd. I never chose that, figured I could install whatever I needed after the fact.

There seems to be two paths one can choose, the Infiniband Support packages from redhat, or install MLNX_OFED_LINUX-5.5-1.0.3.2-rhel7.9-x86_64.iso.

I was successful doing the redhat path and not installing anything from the MLNX iso other than firmware update to the infiniband card.

opensm needs to be running on at least one server on the infiniband network, and then a green link light will be observed on the infiniband cards and it'll all work.

MLNX iso causes removal of some incompatible redhat rpms, where it caused problems gettings nfs+rdma working as well as nfs outright if other servers went doing the redhat "infiniband support" install to get the infiniband connection working.

In the end I concluded go the redhat infiniband support path to get it working, and only use MLNX for firmware update; don't let the MLNX iso install all the stuff in linux unless you know what you are doing and really want all that.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .