Virtual Machine Operation and vMotion Failures on ESXi Hosts Using NVIDIA GPUs

Products

• VMware vSphere ESXi

Issue/Introduction

In the affected environment on ESXi hosts using NVIDIA GPUs, you may experience the following symptoms:

• Virtual machine operations fail, including power operations, migrations, and vMotion.

• The vpxa.log intermittently records the following error:

YYYY-MM-DDTHH:MM:SS.###Z Er(163) Vpxa[2099537]: -->    msg = "Another task is already in progress."

• The hostd.log intermittently records the following error:

YYYY-MM-DDTHH:MM:SS.###Z In(166) Hostd[#######]: [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] HTTP Connection has timed out while waiting for further requests; <io_obj p:0x################, h:-1, <TCP '###.#.#.# : #####'>, <TCP '###.#.#.# : #####'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x################, h:-1, <TCP '###.#.#.# : #####'>, <TCP '###.#.#.# : #####'>>, duration: ##:##:##.###### (hh:mm:ss.us))

• A massive amount of Xid 45 errors are logged in the vmkernel.log within a very short timeframe, often reaching multiple logs per second.

YYYY-MM-DDTHH:MM:SS.###Z In(182) vmkernel: cpu##:#######)NVRM: Xid (PCI:####:##:##): 45, pid='<unknown>', name=<unknown>, Ch ########

• The following log is recorded after that:

YYYY-MM-DDTHH:MM:SS.###Z In(182) vmkernel: cpu89:2097442)PCIPassthru: ####: Disable Domain for device ####:##:##.5
YYYY-MM-DDTHH:MM:SS.###Z In(182) vmkernel: cpu89:2097442)PCIPassthru: ####: pcipdevInfo: ############ (####:##:##.5), state 0, destroyed

Environment

VMware ESXi 8.0

Cause

An exception in the GPU driver layer triggers an Xid 45 event, leading to communication timeouts across host management services.

Resolution

To recover from this issue and obtain a driver-level resolution, perform the following steps:

Workaround
Reboot the affected ESXi.

Contact Vendor
Contact NVIDIA support for further investigation.