Live migration fails for NVIDIA RTX Pro 6000 Blackwell DC VMs with libnvidia-vgx.so error
Article ID: 447101
Updated On:
Products
Issue/Introduction
Unable to perform live migrations for VMs utilizing the NVIDIA RTX Pro 6000 Blackwell DC GPU.
Cold migrations are successful, but subsequent power-on operations for the migrated VMs fails with below error:
failed to load: plugin 'libnvidia-vgx.so'.
Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_rtx_pro_6000_blackwell_dc-6q'.
Module 'PCIPluginLate' power on failed.
The vmware.log has below backtrace
YYYY-MM-DDT12:07:26.143Z Er(02) vmx - vmiop_log: NVOS status 0x40
YYYY-MM-DDT12:07:26.143Z Er(02) vmx - vmiop_log: Assertion Failed at 0x18050964:356
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: 12 frames returned by backtrace
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(_nv011153vgpu+0x35) [0x4d1808adf5]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(+0x993dc) [0x4d180383dc]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(+0xb1964) [0x4d18050964]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libnvidia-vgx.so(+0xb29be) [0x4d180519be]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /usr/lib64/vmware/plugin/libvmx-vmiop.so(+0x3db9) [0x4d17d8fdb9]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /bin/vmx(+0x68e389) [0x4cd2ef9389]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /bin/vmx(+0x31e45b) [0x4cd2b8945b]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /bin/vmx(+0x296783) [0x4cd2b01783]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /bin/vmx(+0x2970fb) [0x4cd2b020fb]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /bin/vmx(+0x28575d) [0x4cd2af075d]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /lib64/libc.so.6(__libc_start_main+0xdd) [0x4d1519243d]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: /bin/vmx(+0x28641a) [0x4cd2af141a]
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: (0x0): Failed to get SM information
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: (0x0): Initialization: Failed to get static information error 2
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: (0x0): Failed to initialize plugin internal data for inst 0 with error 2 (unable to setup host connection state)
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: (0x0): Initialization: plugin internal data init failed error 2
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - vmiop_log: display_init failed for inst: 0
YYYY-MM-DDT12:07:26.144Z Er(02) vmx - VMIOP: Plugin vmiop-display initialization failed: 2
YYYY-MM-DDT12:07:26.144Z In(05) vmx - [msg.vmx.plugin.vmiop.vgpu.failed] Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_rtx_pro_6000_blackwell_dc-6q'.
YYYY-MM-DDT12:07:26.144Z In(05) vmx - Module 'PCIPluginLate' power on failed.
YYYY-MM-DDT12:07:26.144Z In(05) vmx - VMX_PowerOn: ModuleTable_PowerOn = 0
YYYY-MM-DDT12:07:26.144Z In(05) vmx - MigrateSetStateFinished: type=2 new state=MIGRATE_FROM_VMX_FINISHED
YYYY-MM-DDT12:07:26.144Z In(05) vmx - MigrateSetState: Transitioning from state MIGRATE_FROM_VMX_INIT (8) to MIGRATE_FROM_VMX_FINISHED (12).
YYYY-MM-DDT12:07:26.144Z In(05) vmx - Migrate: Caching migration error message list:
YYYY-MM-DDT12:07:26.144Z In(05) vmx - [msg.vmx.plugin.vmiop.vgpu.failed] Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_rtx_pro_6000_blackwell_dc-6q'.
YYYY-MM-DDT12:07:26.144Z In(05) vmx - [msg.moduletable.powerOnFailed] Module 'PCIPluginLate' power on failed.
Environment
VMware vSphere ESXi 8.0.3
NVIDIA RTX Pro 6000 Blackwell DC
NVIDIA vGPU Driver version 580.126.08
Cause
This issue is caused by a known NVIDIA driver defect affecting vGPU migrations for GPUs with a GPU System Processor (GSP) on the installed driver branch.
Resolution
Coordinate with NVIDIA to download the updated vGPU driver package.
Update the vGPU driver on the affected hosts to NVIDIA vGPU Software Version 20 (driver version 595.58.02) or later.
Review any additional known issues on the vendor's documentation prior to performing the upgrade.
Additional Information
Feedback
