"Detected another rshim backend already connected", vSphere installation failures for systems with NVIDIA BlueField-3 DPUs while rshim service is running

Products

• VMware vSphere ESXi

Issue/Introduction

• Installation of vSphere will fail if the rshim service is running on the DPU BMC of one or more BlueField-3 cards.
• The installation screen will show the following screen with generic error message notifying the user of ESXio installation failing:
  
  
  
  
• Installation log file "weasel.log" will show follow error message :
  
  ERROR    0000:25:00: failed to image ESXio: 0000:25:00: Detected another rshim backend already connected, cannot proceed. backend status: RSHIM_REMOTE_BACKEND (2)
ERROR    0000:25:00: failed to image ESXio (progress=1, osStatus=ESXIO_READY (5))

Environment

Platforms with NVIDIA BlueField-3 DPUs that have vSphere Distributed Service Engine enabled.

Cause

Only one rshim backend may be active at a time, and the vSphere installer uses rshim to install ESXio on the DPU.

Resolution

Turn off rshim service running on bmc/usb of the system by following below steps :

• SSH to the DPU BMC of each card in the system and run below commands :
  
  # systemctl stop rshim.service
# systemctl disable rshim.service

This resolution will have to be performed once and will be persistent to reboots. There are no impacts of stopping and disabling this service.