vMotion Failures and Host Unresponsiveness with NVIDIA vGPU - VMware vSphere ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines configured with NVIDIA vGPUs may experience unresponsiveness, disconnects, or power-off events during vMotion operations. This behaviour correlates with specific NVIDIA driver versions and vMotion timeouts, typically presenting as migration timeouts during the memory precopy phase.

We see the following error:

• ESXi host unresponsiveness during or after failed vMotion attempts.
• Inability to power off or shut down virtual machines via the vCenter console
• The following errors were observed in the /var/run/log/vmkernel.log file of the ESXi. 
  

  <Time Stamp> In(182) vmkernel: cpu37:2110687)NVRM: Xid (PCI:0000:86:00): 109, pid=2111064, name=, channel 0x0000006a, errorString CTX SWITCH TIMEOUT, Info 0x4c433d

• Additionally, /var/run/log/hostd.logs of the ESXi may report migration failure during the memory precopy phase:

  VMotionInitiateSrc: Migration failed after VM memory precopy.

Environment

• VMware vSphere ESXi 8.x
• VDI environments using NVIDIA GRID/vGPU drivers

Cause

NVIDIA GPU driver version 580.126.08 fails to return control to the VMware stack during the vMotion notification phase. This results in a migration timeout and a failure to send data, potentially rendering the ESXi host unresponsive.

Resolution

Verify the installed NVIDIA driver version and upgrade the driver to vGPU 20.2 (595.91.02), otherwise Engage your hardware vendor to validate the environment configuration and ensure the driver upgrade is applied correctly.

Workaround: If vMotion issues persist, disable "Passthrough VM DRS Automation" in the Cluster Advanced Settings (Cluster -> Configure -> vSphere DRS) to prevent automatic migrations until the driver issue is addressed.