Changing from passthrough to non-passthrough devices fails to take effect after a reboot

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides a workaround for a known issue that occurs when a variety of specific conditions apply as outlined in the following example:

Prerequisite conditions:

• This issue occurs on either of the following servers:
  • Server with AMD Input/Output Memory Management Unit (IOMMU)
  • Server with Intel Virtualization Technology for Directed Input/Output (VT-d)
• A PCI device is set to passsthrough mode

Actions taken:

1. Swap the passthrough mode PCI device with a non-passthrough mode PCI device (after you perform this step, the expected behavior is for the PCI device inserted in the original passthrough slot to still show as passthrough).
2. Use vSphere Client to change the device from passthrough to non-passthrough.
3. Reboot.

Result:

• In some cases, vSphere Client indicates that the PCI device is still in passthrough mode. More specifically, when you remove a device that has been configured for passthrough and you replace it with a non-identical type of device, it is possible that the change will not take effect. However, in no case does this issue occur when you replace a PCI device with an identical type of PCI device.

Environment

VMware vSphere ESXi 5.1
VMware ESXi 4.1.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.0
VMware ESXi 4.0.x Installable
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.5

Resolution

To avoid this issue, use this best practice:

Always unmark devices for passthrough before removing them from the server.

Workaround

 

To force the change from passthrough mode to non-passthrough mode, complete these steps:

1. Edit the /etc/vmware/esx.conf file by changing the owner of the PCI device from passthru to vmkernel.
   
   Example of the edit:
   
   Change from: /device/000:02.0/owner = "passthru"
   
   Change to: /device/000:02.0/owner = "vmkernel"
2. Reboot the ESXi host.

If the above workaround fails to persist after rebooting the ESXi host, the esx.conf is considered malform. A reinstall will need to be performed.