vGPU Virtual Machine goes unresponsive or fails during live migration
search cancel

vGPU Virtual Machine goes unresponsive or fails during live migration

book

Article ID: 403588

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • VM becomes unresponsive/powers off after the vMotion task is completed or failed.

    • Environment includes Citrix persistent VDI.

  • Log signature of the issue in vmware.log will look like below:

    • YYYY-MM-DDTHH:MM:SS In(PID) vmx - Vix: [mainDispatch.c:1172]: VMAutomationPowerOff: Powering off.

      YYYY-MM-DDTHH:MM:SS  Wa(PID) vmx - /vmfs/volumes/datastore/vmname.vmx: Cannot remove symlink /var/run/vmware/#/###########_##########/configFile: No such file or directory

      YYYY-MM-DDTHH:MM:SS In(PID) vmx - WORKER: asyncOps=6 maxActiveOps=2 maxPending=2 maxCompleted=2

      YYYY-MM-DDTHH:MM:SS In(PID) vmx - Vix: [mainDispatch.c:4211]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1873, success=1 additionalError=0

      YYYY-MM-DDTHH:MM:SS In(PID) vmx - Msg_Post: Error

      YYYY-MM-DDTHH:MM:SS In(PID) vmx - [msg.moduletable.powerOnFailed] Module 'Migrate' power on failed.

      YYYY-MM-DDTHH:MM:SS In(PID) vmx - [msg.vmx.poweron.failed] Failed to start the virtual machine.

      YYYY-MM-DDTHH:MM:SS Wa(PID) vmx - POST(no connection): msg.vmx.poweron.failed

      YYYY-MM-DDTHH:MM:SS Wa(PID)+ vmx - Module 'Migrate' power on failed.

      YYYY-MM-DDTHH:MM:SS Wa(PID)+ vmx - Failed to start the virtual machine.

    • vmkernel.log:

      YYYY-MM-DDTHH:MM:SS In(PID) vmkernel: cpu69:#######)VmMemXfer: vm ######: ####: Evicting VM with path:/vmfs/volumes/datastore/vmname.vmx

      YYYY-MM-DDTHH:MM:SS In(PID) vmkernel: cpu69:#######)VmMemXfer: 211: Creating crypto hash

      YYYY-MM-DDTHH:MM:SS In(PID) vmkernel: cpu69:#######)VmMemXfer: vm ######: ####: Could not find MemXferFS region for /vmfs/volumes/datastore/vmname.vmx

  • vMotion for vGPU-enabled VMs fails at 55% with the following error:
    • Migration to host &It;##.##.###.###> failed with error Connection reset by peer (195887179).
YYYY-MM-DDT04:13:51.735034Z
Failed waiting for data. Error 195887137. Timeout.
YYYY-MM-DDT04:14:10.520495Z&:#10;The migration transfer failed during the receive operation to socket mid: received 0/36 bytes: Connection reset by peer.
vMotion migration [170978486:mid] failed to read stream 
  • Log signature of the issue in vmware.log will look like below:
    • YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="worker-2578622"] VMIOP: registering migrate bufs for devNum=0
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="worker-2578622"] vmiop_log: (0x0): ZeroCopy migration buffers registered
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vthread-2578639"] vmiop_log: (0x0): Start restoring vGPU state ...
      YYYY-MM-DDTHH:MM:SS -ERROR vmx 2578615 [esx@4413 threadName="vthread-2578639"] vmiop_log: (0x0): Encountered a migration data block of unsupported version. Failing.
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vthread-2578639"] vmiop_log: (0x0): Migration Ended
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] Migrate_SetFailure called from MigrateWaitForData:3842.
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] Migrate_SetFailureMsgList called from Migrate_SetFailureImpl:3096.
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] MigrateSetStateFinished: type=2 new state=MIGRATE_FROM_VMX_FINISHED
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] MigrateSetState: Transitioning from state MIGRATE_FROM_VMX_PRECOPY (10) to MIGRATE_FROM_VMX_FINISHED (12).
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] Migrate: Caching migration error message list:
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] [msg.migrate.waitdata.platform] Failed waiting for data.  Error bad0021. Timeout.
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] Migrate: cleaning up migration state.
      YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] MigrateDevStreamCleanup: Waiting for migrateDevStreamGroup to complete.
  • Log signature of the issue in vmkernel.log on the destination host will look like below:
    • YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu72:2573804)VMotionEncryption: 260: 7654138476239577974 D: Data crypto stats: Observed bw: 251 MBps, # segments: 16885
      YYYY-MM-DDTHH:MM:SS -WARNING vmkwarning - [esx@4413] cpu2:2578699) MigrateCpt: 1411: 7654138476327532821 D: Timeout waiting for free recvList node (curSegmentId 1 lastSegmentId 1 recvListTotalSize 6250000)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionRecv: 4574: 7654138476327532821 D: Error handling message: Timeout
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateNet: 4427: 7654138476327532821 D: Stats for socket recvHelper(0x4322b4cdd6f0) write: opCount=12 bytes=696 opTimeUs=195 waitTimeUs=0
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateNet: 4434: 7654138476327532821 D: Stats for socket recvHelper(0x4322b4cdd6f0) read: opCount=598 bytes=73422388 opTimeUs=208216 waitTimeUs=7080351769636
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)Migrate: 100: 7654138476327532821 D: MigrateState: Failed
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2068: 7654138476327532821 D: (uptime,ready,used,wait)ms for stream 0: core(27400,172,1233,25272), crypto(27400,150,1094,24711), comp(27400,148,1112,24590)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2068: 7654138476327532821 D: (uptime,ready,used,wait)ms for stream 1: core(27400,155,1506,24615), crypto(27400,171,1423,23635), comp(27400,169,1405,23492)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2073: 7654138476327532821 D: total (uptime,ready,used,wait)ms for helpers: core(54801,327,2740,49888), crypto(54801,321,2518,48346), comp(54801,318,2518,48083)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2075: 7654138476327532821 D: (uptime,ready,used,wait)ms for send helper: (27400,1,1,27396)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2078: 7654138476327532821 D: (uptime,ready,used,wait)ms for recv helper: (27400,4,46,27211)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2194: 7654138476327532821 D: Encryption stats (encBytes, encTimeMs, decBytes, decTimeMs) for vMotion streams
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2206: 7654138476327532821 D: stream helper 0: core(64,1,71883204,86), crypto(0,1,678567936,175), comp(0,1,765190144,185)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2206: 7654138476327532821 D: stream helper 1: core(64,1,123497228,121), crypto(0,1,986685440,244), comp(0,1,1074176000,255)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2224: 7654138476327532821 D: send helper: (860,1,4972,1) recv helper: (480,1,73411624,102)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2237: 7654138476327532821 D: Total: (1468,1,3773416548,1171)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateLog: 1701: 7654138476327532821 D: Capturing cpu info at precopy end
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateLog: 1704: 7654138476327532821 D: Capturing cpu info at precopy end: 4301251
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateLog: 1786: 7654138476327532821 D: Capturing cpu info at migration end
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateLog: 1789: 7654138476327532821 D: Capturing cpu info at migration end: 4301251
      YYYY-MM-DDTHH:MM:SS -WARNING vmkwarning - [esx@4413] cpu2:2578699) Migrate: 255: 7654138476327532821 D: Failed: Timeout (0xbad0021) @0x42001ba23ef0
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotion: 8387: 7654138476327532821 D: Estimated network bandwidth 172.419 MB/s before failure
      YYYY-MM-DDTHH:MM:SS -WARNING vmkwarning - [esx@4413] cpu45:2578615) Migrate: 7510: 7654138476327532821 D: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu45:2578615)Migrate: 92: 7654138476327532821 D: Caller 0x42001efeeade trying to set state to 9 (Failed) after state already set to Failed!
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu13:2578615)Hbr: 4292: Migration end received (worldID=2578615) (migrateType=1) (event=1) (isSource=0) (sharedConfig=1)
      YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu13:2578615)VMotion: 8617: 7654138476327532821 D: VMotion CleanupMigration

 

 

Environment

VMware vSphere ESXi 8.x

VCF 9.0

Cause

NVIDIA's GPU/driver is causing the timeout and failure to send data, migration process is waiting for GPU checkpoint data from NVIDIA for that duration during the Switchover phase.

Resolution

  • Ensure that the latest NVIDIA driver (GRID 18.5 or newer) is installed. For further queries, reach out to NVIDIA support.

  • Upgrade ESXi hosts to version 8.0 Update 3h (Build 25067014) or later.

  • Workaround:

    • Requirements from vGPU Virtual Machine automated migration for Host Maintenance Mode in a DRS Cluster:

      • Healthy vSphere Cluster Services  (Refer to: vSphere Cluster Services (vCLS) Known Issues/Corner Cases).

      • Healthy GPU and GPU driver (e.g. no Xids or Assertions)

      • Configuration of the VM's vGPU devices through the VCenter UI only.

      • Healthy vMotion network (Example: vMotion NICs setup through Cluster QuickStart).

    • Disable Passthrough VM DRS Automation until NVIDIA's driver issue is fixed (e.g. PassthroughDrsAutomation set to 0). This will reduce the probability of vMotion issues as vGPU VMs will not attempt to migrate.

      • There may by tendency of vMotion issue if an attempt of evacuating a host using Maintenance Mode process. To avoid this, PowerOff vGPU VMs before initiating the Host Maintenance Mode task.

    • If VI Admin wants to continue to use Passthrough VM DRS Automation.

      • For Ampere and newer GPUs with GRID 17.x and newer, a VI Admin can set the configuration option 'vmx.plugin.vmiop.waitForMigrateReadCallback="TRUE"' in the host config /etc/vmware/config or in the VMX configuration to further reduce the probability of the migration failing by disabling parallelized GPU checkpoint on the source.

Additional Information

The vMotion Process Under the Hood

vGPU Virtual Machine automated migration for Host Maintenance Mode in a DRS Cluster

  • Plan A for experiences if the concern is with a 10 second VDI connection timeout and does not need DRS automation:

  • Plan B for experiences if the concern is not with a 10 second VDI connection timeout and does not need DRS automation for Maintenance Mode evacuations only:

  • Plan C for customer experience if they are concerned with a 10 second VDI connection timeout and wants DRS automation only:

  • Plan C will enable DRS automation for Load Balancing and Maintenance Mode migrations. This will only occur if the VM estimates if it can migrate under 9 seconds, assuming your VDI connection timeout is 10 seconds this should avoid most migrations of VDI VMs exceeding 9 seconds (assuming a healthy cluster with sufficient network bandwidth).