Host does not respond after migrating virtual machine
Article ID: 344760
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
1. No host response after a migration task.
2. VM will lose network connection after re-edit the network setting.
3. VM cannot receive packet from the VM on the same host‘s portgroup
4. You will see the error from the hostd log and vpxa log
2018-11-28T14:17:44.107Z info hostd[61443B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx] Tools manifest version status changed from to guestToolsCurrent, on install is FALSE
2018-11-28T14:17:44.108Z info hostd[60E80B70] [Originator@6876 sub=Libs opID=HB-host-26016@472-71027035-c4-ecc1 user=vpxuser] GetHostManifests: Done extracting the manifest file.
2018-11-28T14:17:44.111Z info hostd[60E80B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx opID=HB-host-26016@472-71027035-c4-ecc1 user=vpxuser] Initial tools version: 3:guestToolsCurrent
2018-11-28T14:17:44.131Z info hostd[60E80B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx opID=HB-host-26016@472-71027035-c4-ecc1 user=vpxuser] Initialized virtual machine.
2018-11-28T14:17:44.131Z info hostd[61443B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx] Skip a duplicate transition to: VM_STATE_ON
2018-11-28T14:17:44.131Z info hostd[61443B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx] Send config update invoked
2018-11-28T14:17:54.048Z info hostd[60C40B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx opID=HB-host-26016@485-bd6393d-89-ece6 user=vpxuser] State Transition (VM_STATE_ON -> VM_STATE_RELOADING)
2018-11-28T14:17:54.048Z info hostd[60C40B70] [Originator@6876 sub=Libs opID=HB-host-26016@485-bd6393d-89-ece6 user=vpxuser] VigorTransportClientManageConnection: connection closed.
018-11-28T14:17:54.048Z info hostd[60C40B70] [Originator@6876 sub=Libs opID=HB-host-26016@485-bd6393d-89-ece6 user=vpxuser] VigorTransportClientManageConnection: connection closed.
2018-11-28T14:17:54.050Z info hostd[5FC9FB70] [Originator@6876 sub=Libs] VigorOnlineDisconnectCb: connection closed (is final).
2018-11-28T14:16:36.526Z info vpxa[FFC38AE0] [Originator@6876 sub=vpxLro opID=0BF54ECD-000000ED-eb-b9-17-9c] [VpxLRO] -- BEGIN task-10454 -- -- vim.host.VMotionManager.initiateDestination:tracking --
2018-11-28T14:16:36.526Z verbose vpxa[FFC38AE0] [Originator@6876 sub=vpxaMoVMotion opID=0BF54ECD-000000ED-eb-b9-17-9c] [MIGRATE] (1543385844856275) tracking progress at destination...
2018-11-28T14:16:36.527Z verbose vpxa[54D45B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-521d9836] [WaitForUpdatesDone] Received callback
2018-11-28T14:16:36.527Z verbose vpxa[54D45B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-521d9836] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 4370 to 4371 (at 4370)
2018-11-28T14:16:36.527Z verbose vpxa[54D45B70] [Originator@6876 sub=hostdvm opID=WFU-521d9836] [VpxaHalVmHostagent] 1: Config changed 'config.extraConfig'
2018-11-28T14:16:36.527Z info vpxa[54D45B70] [Originator@6876 sub=hostdhost opID=WFU-521d9836] [VpxaMoHost::QueryOverheadEx] Found file backing info for device 2000 of type vim.vm.device.VirtualDisk, removing vpxd moref vim.Datastore:5bec3c86-c8fbf66e-b023-e4434b2131d0 before passing to hostd
2018-11-28T14:16:36.527Z info vpxa[54D45B70] [Originator@6876 sub=hostdhost opID=WFU-521d9836] [VpxaMoHost::QueryOverheadEx] Found network backing info for device 4000 of type vim.vm.device.VirtualVmxnet3, removing vpxd moref vim.Network:HaNetwork-VM Network(10.110.37.x) before passing to hostd
vpxa log
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=QS-host-26016-525107df-f3] [VpxaHalCnxHostagent::DoCheckForUpdates] CheckForUpdates() done; version sent: 4623, version received: 4625
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=QS-host-26016-525107df-f3] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 4623 to 4625 (at 4624)
2018-11-28T14:21:23.266Z info vpxa[54CE2B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=QS-host-26016-525107df-f3] [VpxaHalCnxHostagent::ProcessUpdate] Received old update set base version 4623 (< 4624), discarding
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=vpxaMoService opID=QS-host-26016-525107df-f3] [VpxaMoService] GetChanges: 802 -> 813
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2+] VpxaStatsMetadata::PrepareStatsChanges
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=vpxaStatsMetadata opID=QS-host-26016-525107df-f3] [VpxaStatsMetadata] Prior to refilling translations (412, 412)
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=vpxaStatsMetadata opID=QS-host-26016-525107df-f3] [VpxaStatsMetadata] No new counters to fill
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2-] VpxaStatsMetadata::PrepareStatsChanges (took 0 ms)
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2+] VpxdDatastore::PrepareChanges
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2-] VpxdDatastore::PrepareChanges (took 0 ms)
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2+] VpxaMoService::PrepareHostChanges
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=vpxaMoService opID=QS-host-26016-525107df-f3] [VpxaMoService] host changes include changed properties []
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2-] VpxaMoService::PrepareHostChanges (took 0 ms)
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2+] VpxaMoService::PrepareVmChanges
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [3+] HandleVMChanges for vmid:2
2. VM will lose network connection after re-edit the network setting.
3. VM cannot receive packet from the VM on the same host‘s portgroup
4. You will see the error from the hostd log and vpxa log
2018-11-28T14:17:44.107Z info hostd[61443B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx] Tools manifest version status changed from to guestToolsCurrent, on install is FALSE
2018-11-28T14:17:44.108Z info hostd[60E80B70] [Originator@6876 sub=Libs opID=HB-host-26016@472-71027035-c4-ecc1 user=vpxuser] GetHostManifests: Done extracting the manifest file.
2018-11-28T14:17:44.111Z info hostd[60E80B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx opID=HB-host-26016@472-71027035-c4-ecc1 user=vpxuser] Initial tools version: 3:guestToolsCurrent
2018-11-28T14:17:44.131Z info hostd[60E80B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx opID=HB-host-26016@472-71027035-c4-ecc1 user=vpxuser] Initialized virtual machine.
2018-11-28T14:17:44.131Z info hostd[61443B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx] Skip a duplicate transition to: VM_STATE_ON
2018-11-28T14:17:44.131Z info hostd[61443B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx] Send config update invoked
2018-11-28T14:17:54.048Z info hostd[60C40B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5bec3c86-c8fbf66e-b023-e4434b2131d0/test_vmotion_withvmtools/test_vmotion_withvmtools.vmx opID=HB-host-26016@485-bd6393d-89-ece6 user=vpxuser] State Transition (VM_STATE_ON -> VM_STATE_RELOADING)
2018-11-28T14:17:54.048Z info hostd[60C40B70] [Originator@6876 sub=Libs opID=HB-host-26016@485-bd6393d-89-ece6 user=vpxuser] VigorTransportClientManageConnection: connection closed.
018-11-28T14:17:54.048Z info hostd[60C40B70] [Originator@6876 sub=Libs opID=HB-host-26016@485-bd6393d-89-ece6 user=vpxuser] VigorTransportClientManageConnection: connection closed.
2018-11-28T14:17:54.050Z info hostd[5FC9FB70] [Originator@6876 sub=Libs] VigorOnlineDisconnectCb: connection closed (is final).
2018-11-28T14:16:36.526Z info vpxa[FFC38AE0] [Originator@6876 sub=vpxLro opID=0BF54ECD-000000ED-eb-b9-17-9c] [VpxLRO] -- BEGIN task-10454 -- -- vim.host.VMotionManager.initiateDestination:tracking --
2018-11-28T14:16:36.526Z verbose vpxa[FFC38AE0] [Originator@6876 sub=vpxaMoVMotion opID=0BF54ECD-000000ED-eb-b9-17-9c] [MIGRATE] (1543385844856275) tracking progress at destination...
2018-11-28T14:16:36.527Z verbose vpxa[54D45B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-521d9836] [WaitForUpdatesDone] Received callback
2018-11-28T14:16:36.527Z verbose vpxa[54D45B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-521d9836] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 4370 to 4371 (at 4370)
2018-11-28T14:16:36.527Z verbose vpxa[54D45B70] [Originator@6876 sub=hostdvm opID=WFU-521d9836] [VpxaHalVmHostagent] 1: Config changed 'config.extraConfig'
2018-11-28T14:16:36.527Z info vpxa[54D45B70] [Originator@6876 sub=hostdhost opID=WFU-521d9836] [VpxaMoHost::QueryOverheadEx] Found file backing info for device 2000 of type vim.vm.device.VirtualDisk, removing vpxd moref vim.Datastore:5bec3c86-c8fbf66e-b023-e4434b2131d0 before passing to hostd
2018-11-28T14:16:36.527Z info vpxa[54D45B70] [Originator@6876 sub=hostdhost opID=WFU-521d9836] [VpxaMoHost::QueryOverheadEx] Found network backing info for device 4000 of type vim.vm.device.VirtualVmxnet3, removing vpxd moref vim.Network:HaNetwork-VM Network(10.110.37.x) before passing to hostd
vpxa log
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=QS-host-26016-525107df-f3] [VpxaHalCnxHostagent::DoCheckForUpdates] CheckForUpdates() done; version sent: 4623, version received: 4625
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=QS-host-26016-525107df-f3] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 4623 to 4625 (at 4624)
2018-11-28T14:21:23.266Z info vpxa[54CE2B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=QS-host-26016-525107df-f3] [VpxaHalCnxHostagent::ProcessUpdate] Received old update set base version 4623 (< 4624), discarding
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=vpxaMoService opID=QS-host-26016-525107df-f3] [VpxaMoService] GetChanges: 802 -> 813
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2+] VpxaStatsMetadata::PrepareStatsChanges
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=vpxaStatsMetadata opID=QS-host-26016-525107df-f3] [VpxaStatsMetadata] Prior to refilling translations (412, 412)
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=vpxaStatsMetadata opID=QS-host-26016-525107df-f3] [VpxaStatsMetadata] No new counters to fill
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2-] VpxaStatsMetadata::PrepareStatsChanges (took 0 ms)
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2+] VpxdDatastore::PrepareChanges
2018-11-28T14:21:23.266Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2-] VpxdDatastore::PrepareChanges (took 0 ms)
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2+] VpxaMoService::PrepareHostChanges
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=vpxaMoService opID=QS-host-26016-525107df-f3] [VpxaMoService] host changes include changed properties []
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2-] VpxaMoService::PrepareHostChanges (took 0 ms)
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [2+] VpxaMoService::PrepareVmChanges
2018-11-28T14:21:23.267Z verbose vpxa[54CE2B70] [Originator@6876 sub=VpxProfiler opID=QS-host-26016-525107df-f3] [3+] HandleVMChanges for vmid:2
Environment
VMware vSphere ESXi 6.0
Cause
When host using ixgben driver, vpxa and hostd might have pending callbacks. It may cause the management service (hostd) to stop responding.
Resolution
This is a known issue affecting VMware ESXi 6.0
This issues is resolved in VMware ESXi 6.0 P07
This issues is resolved in VMware ESXi 6.0 P07
Workaround:
Restart the management service by DCUI.
or
Using ixgbe driver instead of ixgben.
or
Using ixgbe driver instead of ixgben.
Feedback
Yes
No
Powered by
