VM poweron fails due to hotplug bus reset after adding a third NVIDIA passthru device

Products

• VMware vSphere ESXi

Issue/Introduction

• Powering on a VM fails after adding a third NVIDIA passthrough GPU device.
• You observing the following in the /vmfs/volumes/<datastore>/<vmname>/vmware.log :

[YYYY-MM-DDTHH:MM:SS] Wa(03) vmx - PCIPassthru: Selected device 0000:##:00.0 is outside of the NUMA configuration
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - PCIPassthru: Failed to register device 0000:da:00.0  error = 0xffffffff
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - Msg_Post: Error
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - [msg.pciPassthru.createAdapterFailedPlatformNotSupported] Failed to register the device pciPassthru0 for 102:##.# due to unavailable hardware or software support.

[YYYY-MM-DDTHH:MM:SS] In(05)+ vmx - Power on failure messages: Module 'DevicePowerOn' power on failed.
[YYYY-MM-DDTHH:MM:SS] In(05)+ vmx - Failed to register the device pciPassthru0 for 102:##.# due to unavailable hardware or software support.
[YYYY-MM-DDTHH:MM:SS] In(05)+ vmx - Failed to start the virtual machine.

• Following entries are found in /var/run/log/vmkernel.log :

[YYYY-MM-DDTHH:MM:SS] cpu8:2097563)PCIEHP: 1477: 0000:##:00.0: hotplug slot:0x19 (0000:##:00.0) Adapter inserted.
[YYYY-MM-DDTHH:MM:SS] cpu8:2097563)PCIEHP: 380: 0000:##:00.0: hotplug slot:0x19: Setting PowerIndicator State BLINKING
[YYYY-MM-DDTHH:MM:SS] cpu8:2097564)PCIEHP: 1564: 0000:##:00.0: hotplug slot:0x19: num reads=1 slot status=0x50.
[YYYY-MM-DDTHH:MM:SS] cpu5:2097563)PCIEHP: 982: 0000:##:00.0: Enabling hotplug slot:0x19
[YYYY-MM-DDTHH:MM:SS] cpu5:2097563)WARNING: PCIEHP: 641: 0000:##:00.0: hotplug slot: 0x19: Device insertion detected while prior device 0000:##:00.0 removal is still pending
[YYYY-MM-DDTHH:MM:SS] cpu52:2102781)PCIPassthru: 1496: Disable Domain for device 0000:##:00.0
[YYYY-MM-DDTHH:MM:SS] cpu52:2102781)PCIPassthru: 847: pcipdevInfo: 0x431075001550 (0000:##:00.0), state 0, destroyed
[YYYY-MM-DDTHH:MM:SS] cpu30:2098449)PCIPassthru: PCIPassthruForgetDev:546: Forgot device 0000:##:00.0
[YYYY-MM-DDTHH:MM:SS] cpu30:2098449)Device: 766: PCIPassthru:driver->ops.forgetDevice:6006 ms
[YYYY-MM-DDTHH:MM:SS] cpu30:2098449)PCIPassthru: PCIPassthruQuiesceDev:445: Quiesced device 0000:##:00.0
[YYYY-MM-DDTHH:MM:SS] cpu30:2098449)Device: 646: PCIPassthru:driver->ops.quiesceDevice:0 ms
[YYYY-MM-DDTHH:MM:SS] cpu30:2098449)PCIPassthru: PCIPassthruDetachDev:361: Device 0000:##:00.0 removed from passthru sbdfmap(0x4310750012a0)
[YYYY-MM-DDTHH:MM:SS] cpu30:2098449)PCIPassthru: PCIPassthruDetachDev:364: Detached passthru device 0000:##:00.0
[YYYY-MM-DDTHH:MM:SS] cpu30:2098449)Device: 460: PCIPassthru:driver->ops.detachDevice:0 ms
[YYYY-MM-DDTHH:MM:SS] cpu30:2098449)Device: 1755: Unregistered device: 0x430ad5003880 s00000019.00 10de223510de145a030200
[YYYY-MM-DDTHH:MM:SS] cpu30:2098449)Device: 724: pci:ops->removeDevice:0 ms
[YYYY-MM-DDTHH:MM:SS] cpu52:2102781)PCIPassthru: 4495: 0000:##:00.0: device not found


Cause

The issue occurs because a hotplug bus reset triggers a reset of the adapter.

Resolution

• There are 2 options to workaround this hotplug bus failure :

1. 1. Work with your hardware vendor to disable hotplug on the PCIe slot.
   2. Modify the file /etc/vmware/passthru.map

To modify /etc/vmware/passthru.map, follow the below steps :

1. 1. 1. 1. Modify the file /etc/vmware/passthru.map and change the value from "bridge" to "flr" :
            

            # NVIDIA (FLR issue on Ampere and Hopper GPUs)
10de ffff bridge false

            Change to:

            # NVIDIA (FLR issue on Ampere and Hopper GPUs)
10de ffff flr false

         2. Reboot the host.

 

Additional Information

This only affects NVIDIA GPUs when the card is slotted into a PCIe slot configured with Hotplug.