vMotion Involving VMs with NVIDIA vGPU hang at 68% after upgrading ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

• vMotion Involving VMs with NVIDIA vGPU configurations have failed after upgrading ESXi host
• vMotions were successful prior to upgrading the ESXi hosts
• vGPU Manager was also upgraded on the upgraded ESXi hosts
• There are no NVRM: Xid messages in the VM's vmware.log files
• When the VM is being vMotioned and is still on the source host, the VM's vmware.log files contain:

<TIMESTAMP> In(05) vcpu-0 - Migrate_Open: Migrating to <SOURCE_ESXI_HOST> with migration id <MIGRATION_ID>
<TIMESTAMP> In(05) vcpu-0 - Progress -1% (msg.checkpoint.saveStatus)
<TIMESTAMP> In(05) vcpu-0 - Checkpointed in VMware ESX, 8.0.3, build-24859861, Linux Host
<TIMESTAMP> In(05) vthread-40866634 - vmiop_log: (0x0): Start saving vGPU state ...
<TIMESTAMP> In(05) vthread-40866634 - vmiop_log: (0x0): Finish saving vGPU state ...
<TIMESTAMP> In(05) vcpu-0 - DeviceIfCheckPointMigrate 0000:46:00.5 SAVING: DevCptReadBytes 59686824 ret=3
<TIMESTAMP> In(05) worker-40866650 - MigrateDevCptStreamWorker: Finish transmitting device checkpoint data. (endOfStreams true failureCode 0 isListRetired false)

• When the vCenter migrate task is at 68%, the destination host is attempting to resume the VM and the VM's vmware.log file contain:
  
  <TIMESTAMP> In(05) vmx - vmiop_log: Driver Version: 580.126.08
  <TIMESTAMP> In(05) worker-2101709 - MigrateDevCptStreamWorker: Finish writing device checkpoint data to device. (endOfStreams true failureCode 0 isListRetired false)
  <TIMESTAMP> In(05) vthread-2101717 - vmiop_log: Source host driver version: 535.274.03
  <TIMESTAMP> In(05) vthread-2101717 - vmiop_log: (0x0): vGPU UUID : <VGPU_UUID>
  <TIMESTAMP> Er(02) vthread-2101717 - vmiop_log: (0x0): Unsupported block header version (0x0) of migration data encountered
  <TIMESTAMP> In(05) vthread-2101717 - vmiop_log: (0x0): Migration Ended
  <TIMESTAMP> In(05) vmx - MigrateWaitForData: Waited for 10.42 seconds.

Environment

ESXi 8.0.3

Cause

The vmware.log shows the destination source driver version as 580.126.08 and the source host driver version as 535.274.03

According to the NVIDIA documentation (https://docs.nvidia.com/vgpu/19.0/known-issues/bug-200602087-suspend-resume-different-vgpu-manager-versions-fails.html), these are not compatible:

“Suspending a VM configured with vGPU on a host running one version of the vGPU manager and resuming the VM on a host running a version from an older main release branch fails. For example, suspending a VM on a host that is running the vGPU manager from release 19.5 and resuming the VM on a host running the vGPU manager from release 18.6 fails.”

This incompatibility is what causes “vmiop_log: (0x0): Unsupported block header version (0x0) of migration data encountered”.

Resolution

Contact NVIDIA support troubleshooting this issue. If NVIDIA needs to collaborate with Broadcom support, create a Broadcom support case and reference NVIDIA's case number.