Configuring NVIDIA BlueField-3 (BF3) into a "DPU mode for VMware vSphere Distributed Services Engine" OR "NIC mode for vSphere"
search cancel

Configuring NVIDIA BlueField-3 (BF3) into a "DPU mode for VMware vSphere Distributed Services Engine" OR "NIC mode for vSphere"

book

Article ID: 379391

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

NVIDIA BlueField-3 Modes of Operation applicable for VMware vSphere:

  1. DPU mode

    DPU mode, or embedded function (ECPF) ownership, where the embedded Arm system controls the NIC resources and data path. This is the default mode for the BlueField DPU. In DPU mode, INTERNAL_CPU_OFFLOAD_ENGINE is ENABLED(0). vSphere Distributed Services Engine feature requires BlueField DPU is in DPU mode with VMware Distributed Services Engine (DPU) enabled in device settings.

  2. NIC mode

    NIC mode, where BlueField behaves exactly like an adapter card from the perspective of the external host. The ECPFs (embedded cpu function) on the Arm side are not functional in this mode but the user is still able to access the Arm system and update mlxconfig options. In NIC mode, INTERNAL_CPU_OFFLOAD_ENGINE  is DISABLED(1).  When BlueField DPU is configured in NIC mode, ESXi detects it as adapter card - a regular NIC.

    Refer NVIDIA BlueField Modes of Operation for further details on various BlueField Modes of Operation

Environment

vSphere 8.0 U3b and higher versions.

Resolution

  • Configuring NVIDIA BlueField-3 (BF3) for VMware Distributed Services Engine (DPU)

    For configuring brand new BlueField-3 DPU out of factory for vSphere Distributed Services Engine, enable "VMware Distributed Services Engine (DPU)" under BlueField3 device settings from server BIOS/UEFI setup.

    1. Boot the Server into BIOS setup.
    2. Navigate to Device Settings menu.
    3. Select NVIDIA BlueField-3 (BF3) network adapter device entries.
    4. Under BF3 Network Adapter Main Config page, select "Enabled" option for "VMware Distributed Services Engine (DPU)".


      Note: You will notice VMware Configuration option shows up at adapter config page.

    5. Repeat steps (3) & (4) for all the BF3 adapters.
    6. Save and exit followed by Power Cycle (Cold Boot) of the Server.

  • Configuring NVIDIA BlueField-3 (BF3) into a NIC mode

    BlueField-3 DPU out of the factory, by default is in the DPU mode and VMware Distributed Services Engine (DPU) is disabled.

    Perform following steps to configure BF3 into NIC mode to use it as regular network adapter card - a regular NIC:
    • Refer NIC Mode for BlueField-3 sections at NVIDIA BlueField Modes of Operation from NVIDIA for various available methods to configure BF3 into NIC mode on a brand new BF3.
    • On a BF3 with ESXi installed and configured, perform the following steps to configure BF3 into a NIC mode
      1. Check and ensure VMware Distributed Services Engine (DPU) is disabled under BlueField-3 network adapter device settings (Server BIOS Setup).
        1. Boot the Server into BIOS setup.
        2. Navigate to Device Settings menu.
        3. Select NVIDIA BlueField-3 (BF3) network adapter device entries.
        4. Under BF3 Network Adapter Main Config page, ensure you see VMware Distributed Services Engine (DPU) option is Disabled. If it is Enabled, disable it.
        5. Repeat steps (3) & (4) for all the BF3 adapters.
        6. Save and exit followed by Power Cycle (Cold Boot) of the Server.
      2. Download and install MFT tools for VMware ESX Server and disable the offload engine:
        1. You can download required MFT tools from NVIDIA Firmware Tools (MFT).
        2. Install Mellanox-NATIVE-NMST and Mellanox-MFT-Tools components on ESXi host and reboot:

          # esxcli software component apply -d /tmp/Mellanox-NATIVE-NMST_*.zip --no-sig-check
          # esxcli software component apply -d /tmp/Mellanox-MFT-Tools_*.zip --no-sig-check
          # reboot

        3. Run following command from ESXi host to disable offload engine to enable NIC mode from DPU mode :

          # /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 set INTERNAL_CPU_OFFLOAD_ENGINE=1

      3. Power Cycle (Cold Boot) the ESXi host.

  • Switching BlueField-3 (BF3) from NIC mode to VMware Distributed Services Engine enabled DPU mode

    On a BF3 DPU configured in NIC mode and ESXi installed, perform the following steps to configure BF3 into a DPU mode with VMware Distributed Services Engine (DPU) enabled.
    1. Check and enable the VMware Distributed Services Engine (DPU) under BlueField-3 network adapter device settings at Server BIOS Setup using the procedure at "2. Configuring BlueField-3 (BF3) for VMware Distributed Services Engine (DPU)"
    2. Download and install MFT tools for VMware ESX Server and enable the offload engine:
      1. You can download required MFT tools from NVIDIA Firmware Tools (MFT).
      2. Install Mellanox-NATIVE-NMST and Mellanox-MFT-Tools components on ESXi host and reboot:

        # esxcli software component apply -d /tmp/Mellanox-NATIVE-NMST_*.zip --no-sig-check
        # esxcli software component apply -d /tmp/Mellanox-MFT-Tools_*.zip --no-sig-check
        # reboot

      3. Run following command from ESXi host to enable offload engine to enable DPU mode from NIC mode:

        # /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 set INTERNAL_CPU_OFFLOAD_ENGINE=0

      4. Power Cycle (Cold Boot) the ESX host.

        IMPORTANT NOTE:  When you switch BlueField-3 from NIC mode to VMware Distributed Services Engine enabled DPU mode, you must re-install ESXi and reconfigure the host.

  • Switching BlueField-3 (BF3) from VMware Distributed Services Engine enabled DPU mode to NIC mode

    On a BF3 DPU with VMware Distributed Services Engine (DPU) enabled and ESXi installed, perform the following steps to configure BF3 into a NIC mode.
    1. Check and disable the VMware Distributed Services Engine (DPU) under BlueField-3 network adapter device settings at Server BIOS Setup
      1. Boot the Server into BIOS setup.
      2. Navigate to Device Settings menu.
      3. Select NVIDIA BlueField-3 (BF3) network adapter device entries.
      4. Under BF3 Network Adapter Main Config page, select "Disabled" option for "VMware Distributed Services Engine (DPU)".
      5. Repeat steps (3) & (4) for all the BF3 adapters.
      6. Save and exit followed by Power Cycle (Cold Boot) of the Server.
    2. Download and install MFT tools for VMware ESX Server and disable the offload engine:
      1. You can download required MFT tools from NVIDIA Firmware Tools (MFT).
      2. Install Mellanox-NATIVE-NMST and Mellanox-MFT-Tools components on ESXi host and reboot:

        # esxcli software component apply -d /tmp/Mellanox-NATIVE-NMST_*.zip --no-sig-check
        # esxcli software component apply -d /tmp/Mellanox-MFT-Tools_*.zip --no-sig-check
        # reboot

      3. Run following command from ESXi host to disable offload engine to enable NIC mode from DPU mode :

        # /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 set INTERNAL_CPU_OFFLOAD_ENGINE=1

    3. Power Cycle (Cold Boot) the ESX host.

      IMPORTANT NOTE: When you switch BlueField-3 from VMware Distributed Services Engine enabled DPU mode to NIC mode, you must re-install ESXi and reconfigure the host.

Powered by Wolken Software