VM becomes unresponsive/powers off after the vMotion task is completed or failed.
Environment includes Citrix persistent VDI.
Log signature of the issue in vmware.log will look like below:
YYYY-MM-DDTHH:MM:SS In(PID) vmx - Vix: [mainDispatch.c:1172]: VMAutomationPowerOff: Powering off.
YYYY-MM-DDTHH:MM:SS Wa(PID) vmx - /vmfs/volumes/datastore/vmname.vmx: Cannot remove symlink /var/run/vmware/#/###########_##########/configFile: No such file or directory
YYYY-MM-DDTHH:MM:SS In(PID) vmx - WORKER: asyncOps=6 maxActiveOps=2 maxPending=2 maxCompleted=2
YYYY-MM-DDTHH:MM:SS In(PID) vmx - Vix: [mainDispatch.c:4211]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1873, success=1 additionalError=0
YYYY-MM-DDTHH:MM:SS In(PID) vmx - Msg_Post: Error
YYYY-MM-DDTHH:MM:SS In(PID) vmx - [msg.moduletable.powerOnFailed] Module 'Migrate' power on failed.
YYYY-MM-DDTHH:MM:SS In(PID) vmx - [msg.vmx.poweron.failed] Failed to start the virtual machine.
YYYY-MM-DDTHH:MM:SS Wa(PID) vmx - POST(no connection): msg.vmx.poweron.failed
YYYY-MM-DDTHH:MM:SS Wa(PID)+ vmx - Module 'Migrate' power on failed.
YYYY-MM-DDTHH:MM:SS Wa(PID)+ vmx - Failed to start the virtual machine.
vmkernel.log:
YYYY-MM-DDTHH:MM:SS In(PID) vmkernel: cpu69:#######)VmMemXfer: vm ######: ####: Evicting VM with path:/vmfs/volumes/datastore/vmname.vmx
YYYY-MM-DDTHH:MM:SS In(PID) vmkernel: cpu69:#######)VmMemXfer: 211: Creating crypto hash
YYYY-MM-DDTHH:MM:SS In(PID) vmkernel: cpu69:#######)VmMemXfer: vm ######: ####: Could not find MemXferFS region for /vmfs/volumes/datastore/vmname.vmx
Migration to host &It;##.##.###.###> failed with error Connection reset by peer (195887179). YYYY-MM-DDT04:13:51.735034Z Failed waiting for data. Error 195887137. Timeout. YYYY-MM-DDT04:14:10.520495Z&:#10;The migration transfer failed during the receive operation to socket mid: received 0/36 bytes: Connection reset by peer. vMotion migration [170978486:mid] failed to read stream vmware.log will look like below:YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="worker-2578622"] VMIOP: registering migrate bufs for devNum=0YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="worker-2578622"] vmiop_log: (0x0): ZeroCopy migration buffers registeredYYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vthread-2578639"] vmiop_log: (0x0): Start restoring vGPU state ...YYYY-MM-DDTHH:MM:SS -ERROR vmx 2578615 [esx@4413 threadName="vthread-2578639"] vmiop_log: (0x0): Encountered a migration data block of unsupported version. Failing.YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vthread-2578639"] vmiop_log: (0x0): Migration EndedYYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] Migrate_SetFailure called from MigrateWaitForData:3842.YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] Migrate_SetFailureMsgList called from Migrate_SetFailureImpl:3096.YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] MigrateSetStateFinished: type=2 new state=MIGRATE_FROM_VMX_FINISHEDYYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] MigrateSetState: Transitioning from state MIGRATE_FROM_VMX_PRECOPY (10) to MIGRATE_FROM_VMX_FINISHED (12).YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] Migrate: Caching migration error message list:YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] [msg.migrate.waitdata.platform] Failed waiting for data. Error bad0021. Timeout.YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] Migrate: cleaning up migration state.YYYY-MM-DDTHH:MM:SS -INFO vmx 2578615 [esx@4413 threadName="vmx"] MigrateDevStreamCleanup: Waiting for migrateDevStreamGroup to complete.vmkernel.log on the destination host will look like below:YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu72:2573804)VMotionEncryption: 260: 7654138476239577974 D: Data crypto stats: Observed bw: 251 MBps, # segments: 16885YYYY-MM-DDTHH:MM:SS -WARNING vmkwarning - [esx@4413] cpu2:2578699) MigrateCpt: 1411: 7654138476327532821 D: Timeout waiting for free recvList node (curSegmentId 1 lastSegmentId 1 recvListTotalSize 6250000)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionRecv: 4574: 7654138476327532821 D: Error handling message: TimeoutYYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateNet: 4427: 7654138476327532821 D: Stats for socket recvHelper(0x4322b4cdd6f0) write: opCount=12 bytes=696 opTimeUs=195 waitTimeUs=0YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateNet: 4434: 7654138476327532821 D: Stats for socket recvHelper(0x4322b4cdd6f0) read: opCount=598 bytes=73422388 opTimeUs=208216 waitTimeUs=7080351769636YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)Migrate: 100: 7654138476327532821 D: MigrateState: FailedYYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2068: 7654138476327532821 D: (uptime,ready,used,wait)ms for stream 0: core(27400,172,1233,25272), crypto(27400,150,1094,24711), comp(27400,148,1112,24590)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2068: 7654138476327532821 D: (uptime,ready,used,wait)ms for stream 1: core(27400,155,1506,24615), crypto(27400,171,1423,23635), comp(27400,169,1405,23492)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2073: 7654138476327532821 D: total (uptime,ready,used,wait)ms for helpers: core(54801,327,2740,49888), crypto(54801,321,2518,48346), comp(54801,318,2518,48083)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2075: 7654138476327532821 D: (uptime,ready,used,wait)ms for send helper: (27400,1,1,27396)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2078: 7654138476327532821 D: (uptime,ready,used,wait)ms for recv helper: (27400,4,46,27211)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2194: 7654138476327532821 D: Encryption stats (encBytes, encTimeMs, decBytes, decTimeMs) for vMotion streamsYYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2206: 7654138476327532821 D: stream helper 0: core(64,1,71883204,86), crypto(0,1,678567936,175), comp(0,1,765190144,185)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2206: 7654138476327532821 D: stream helper 1: core(64,1,123497228,121), crypto(0,1,986685440,244), comp(0,1,1074176000,255)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2224: 7654138476327532821 D: send helper: (860,1,4972,1) recv helper: (480,1,73411624,102)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotionStream: 2237: 7654138476327532821 D: Total: (1468,1,3773416548,1171)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateLog: 1701: 7654138476327532821 D: Capturing cpu info at precopy endYYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateLog: 1704: 7654138476327532821 D: Capturing cpu info at precopy end: 4301251YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateLog: 1786: 7654138476327532821 D: Capturing cpu info at migration endYYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)MigrateLog: 1789: 7654138476327532821 D: Capturing cpu info at migration end: 4301251YYYY-MM-DDTHH:MM:SS -WARNING vmkwarning - [esx@4413] cpu2:2578699) Migrate: 255: 7654138476327532821 D: Failed: Timeout (0xbad0021) @0x42001ba23ef0YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu2:2578699)VMotion: 8387: 7654138476327532821 D: Estimated network bandwidth 172.419 MB/s before failureYYYY-MM-DDTHH:MM:SS -WARNING vmkwarning - [esx@4413] cpu45:2578615) Migrate: 7510: 7654138476327532821 D: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error.YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu45:2578615)Migrate: 92: 7654138476327532821 D: Caller 0x42001efeeade trying to set state to 9 (Failed) after state already set to Failed!YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu13:2578615)Hbr: 4292: Migration end received (worldID=2578615) (migrateType=1) (event=1) (isSource=0) (sharedConfig=1)YYYY-MM-DDTHH:MM:SS -INFO vmkernel - [esx@4413] cpu13:2578615)VMotion: 8617: 7654138476327532821 D: VMotion CleanupMigration
VMware vSphere ESXi 8.x
VCF 9.0
NVIDIA's GPU/driver is causing the timeout and failure to send data, migration process is waiting for GPU checkpoint data from NVIDIA for that duration during the Switchover phase.
Ensure that the latest NVIDIA driver (GRID 18.5 or newer) is installed. For further queries, reach out to NVIDIA support.
Upgrade ESXi hosts to version 8.0 Update 3h (Build 25067014) or later.
Workaround:
Requirements from vGPU Virtual Machine automated migration for Host Maintenance Mode in a DRS Cluster:
Healthy vSphere Cluster Services (Refer to: vSphere Cluster Services (vCLS) Known Issues/Corner Cases).
Healthy GPU and GPU driver (e.g. no Xids or Assertions)
Configuration of the VM's vGPU devices through the VCenter UI only.
Healthy vMotion network (Example: vMotion NICs setup through Cluster QuickStart).
Disable Passthrough VM DRS Automation until NVIDIA's driver issue is fixed (e.g. PassthroughDrsAutomation set to 0). This will reduce the probability of vMotion issues as vGPU VMs will not attempt to migrate.
There may by tendency of vMotion issue if an attempt of evacuating a host using Maintenance Mode process. To avoid this, PowerOff vGPU VMs before initiating the Host Maintenance Mode task.
If VI Admin wants to continue to use Passthrough VM DRS Automation.
For Ampere and newer GPUs with GRID 17.x and newer, a VI Admin can set the configuration option 'vmx.plugin.vmiop.waitForMigrateReadCallback="TRUE"' in the host config /etc/vmware/config or in the VMX configuration to further reduce the probability of the migration failing by disabling parallelized GPU checkpoint on the source.
The vMotion Process Under the Hood
vGPU Virtual Machine automated migration for Host Maintenance Mode in a DRS Cluster
Plan A for experiences if the concern is with a 10 second VDI connection timeout and does not need DRS automation:
Disable "Passthrough VM DRS Automation" in the Cluster Advanced Settings (Cluster -> Configure -> vSphere DRS).
See KB: vGPU Virtual Machine automated migration for Host Maintenance Mode in a DRS Cluster for more information.
Plan B for experiences if the concern is not with a 10 second VDI connection timeout and does not need DRS automation for Maintenance Mode evacuations only:
Disable "Passthrough VM DRS Automation" in the Cluster Advanced Settings (Cluster -> Configure -> vSphere DRS).
Add option "VgpuMMAutomationTimeoutSecs" and value "-1" (both without quotes) in the Cluster Advanced Options
See KB: vGPU Virtual Machine automated migration for Host Maintenance Mode in a DRS Cluster for more information.
Plan C for customer experience if they are concerned with a 10 second VDI connection timeout and wants DRS automation only:
Enable "Passthrough VM DRS Automation" in the Cluster Advanced Settings (Cluster -> Configure -> vSphere DRS).
Sets "VM Devices Stun Time Limit" to 9 seconds.
See KB: vGPU Virtual Machine automated migration for Host Maintenance Mode in a DRS Cluster for more information.
Plan C will enable DRS automation for Load Balancing and Maintenance Mode migrations. This will only occur if the VM estimates if it can migrate under 9 seconds, assuming your VDI connection timeout is 10 seconds this should avoid most migrations of VDI VMs exceeding 9 seconds (assuming a healthy cluster with sufficient network bandwidth).