GPU shows Shared Direct or Basic and will not enter passthrough on ESXi
search cancel

GPU shows Shared Direct or Basic and will not enter passthrough on ESXi

book

Article ID: 445893

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • You see one or more PCI passthrough devices, commonly NVIDIA GPUs, showing Active Type "Shared Direct" or "Basic" in the vSphere Client Graphics Devices view, while identical sibling devices show "Direct".

  • You cannot set the affected device to passthrough from the vSphere Client. Toggling passthrough off and on, with reboots, does not change the result.

  • In localcli hardware pci list, the affected device shows a mismatch between its configured and current owner:

    Configured Owner: VM Passthru
    Current Owner: VMkernel
    Module Name: None
    
  • After the host boots, the passthrough module claims the sibling devices and skips the affected device, with no related error. In /var/run/log/vmkernel.log you see the passthrough module attach the working devices and not the affected one:

    PCIPassthru: PCIPassthruAttachDev:233: Attached to device 0000:17:00.0
    PCIPassthru: PCIPassthruAttachDev:233: Attached to device 0000:65:00.0
    
  • An unrelated PCI device in a different slot may appear under passthrough unexpectedly (Current Owner: VM Passthru) even though it is configured for the VMkernel.

Additional symptoms reported:

  • A GPU appeared as missing, then reappeared after a reboot in a different state.
  • The device was suspected to be a hardware fault, although hardware health checks were clean.

Environment

  • VMware ESXi 7.0 and VMware ESXi 8.0 hosts using PCI passthrough (DirectPath I/O), commonly with NVIDIA GPUs.
  • Frequently observed after physical hardware service, such as a card being reseated, replaced, or moved to a different slot, or after a change to Single Root I/O Virtualization (SR-IOV) configuration.

Cause

ESXi records each PCI device's passthrough assignment in the host configuration store (ConfigStore), under component esx, group hardware, key pci_devices, keyed by the device's PCI address. Each record includes a device-layer bus address (dl_bus_address) that ties the assignment to a specific physical slot.

The device-layer bus address is a vital field. It is written when the record is first created and is preserved across later edits. Enabling or disabling passthrough updates only the owner, not the device-layer bus address.

When the stored device-layer bus address no longer matches the slot the device actually occupies, for example after a card is moved or device enumeration shifts during hardware service, ESXi applies the passthrough assignment at boot to whichever device sits at the stored address. The intended device is left owned by the VMkernel, and an unrelated device at the stored address can be pulled into passthrough. Because the device-layer bus address is preserved across edits, toggling passthrough off and on does not correct it.

Resolution

  1. In the vSphere Client, place the host in maintenance mode, then enable SSH on the host: Host > Actions > Services > Enable Secure Shell (SSH). Open an SSH session to the host.

  2. Identify the affected device's PCI address and its real device-layer bus address. Run localcli hardware pci list and note the device's Device Layer Bus Address and Physical Slot.

  3. Read the stored passthrough record and compare its device-layer bus address to the live value from step 2:
    configstorecli config current get -c esx -g hardware -k pci_devices -i <pci-address>
    
    A stored dl_bus_address that points to a different slot than the device occupies confirms this issue.

  4. Delete the stale record so ESXi regenerates it from the live hardware. Also delete any leftover SR-IOV virtual-function records for the same device:
    configstorecli config current delete -c esx -g hardware -k pci_devices -i <pci-address>
    
    Use -i with the specific PCI address on every delete. Do not use --all, which removes every record in the key!

  5. Confirm that only the intended records were removed, and that the other devices' records remain:
    configstorecli config current get -c esx -g hardware -k pci_devices
    
  6. Reboot the host.

  7. After the reboot, confirm the record regenerated with the correct device-layer bus address:
    configstorecli config current get -c esx -g hardware -k pci_devices -i <pci-address>
    
  8. Enable passthrough on the device:
    esxcli hardware pci pcipassthru set -d <pci-address> -e true
    
  9. Reboot the host again.

  10. Verify the result. localcli hardware pci list shows the device with Current Owner: VM Passthru and Module Name: pciPassthru. For a GPU, Configure > Hardware > Graphics > Graphics Devices shows Active Type: Direct.

Additional Information

  • Capture a host support bundle before and after editing the ConfigStore in case you end up needing support.
  • Direct ConfigStore edits are sensitive. Scope every delete with -i and a specific PCI address, and confirm scope with the read in step 5 before rebooting.
  • If the host runs full-device passthrough only and has no vGPU workload, consider removing the manufacturer's host driver so it cannot claim an unreserved GPU.
  • If the device remains owned by the VMkernel after the record regenerates with the correct address, or if the stored address is still incorrect after regeneration, open a support case for further review. See Creating and managing Broadcom cases