vCenter Server Management Plane Unresponsiveness and Host Disconnections During Cluster vMotion Operations
Article ID: 446842
Updated On:
Products
Issue/Introduction
Symptoms
During active cluster rebalancing, automated/manual Distributed Resource Scheduler (DRS) compute vMotion migrations, or high-load provisioning tasks, you observe the following symptoms:
- The vCenter Server management plane becomes temporarily unresponsive or encounters a multi-minute processing freeze, typically during a virtual machine's memory cutover stun window.
- One or more ESXi nodes momentarily display a greyed-out status or disconnect from inventory, logging a transient vmodl.fault.HostCommunication timeout error in vCenter
/var/log/vmware/vpxd/vpxd.log.<DATE>T03:07:53.881Z info vpxd[<07#78>] [Originator@6876 sub=vpxLro opID=<Cd...r-01>] [VpxLRO] -- BEGIN task-<54...6679> -- <JG...VC02> -- Drm.ExecuteVMotionLRO --<DATE>T03:10:27.384Z warning vpxd[<06#55>] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=<HB...1732>] [CreateOrUpdate] Policy with issuer alias https://<JG...rnal>/openidconnect/vsphere.local already exists on host <de...rnal>. Exception: N4Vpxd7Clients22AlreadyExistsExceptionE(Error:<DATE>T03:10:30.128Z error vpxd[<07#78>] [Originator@6876 sub=VmProv opID=<Cd...0101>] Local-VC Host Migrate failed at vpx.vmprov.InitiateSource for poweredOn VM '<JG...VC02>' (<vm...8647>, ds:///vmfs/volumes/<65...ca30>/<JG...VC08>/<JG...VC08>.vmx) on <ho...8215> (<SOURCE_HOST_IP>) in pool <re...8108> with ds ds:///vmfs/volumes/<65...ca30>/ to <ho...4127> (<TARGET_HOST_IP>) in pool <re...8108> with ds ds:///vmfs/volumes/<65...ca30>/ with migId <90...0184> with fault vmodl.fault.HostCommunication: as Operation: Local-VC_DRS_NonMM_ComputevMotion<DATE>T03:10:33.938Z info vpxd[<06#00>] [Originator@6876 sub=InvtVm opID=<HB...4b50>] VMOTION_RECOVER - Stealing VM <JG...VC02> (ID <vm...8647>) from host <pr...rnal> to host <jg...rnal>; reason: VM is gone from source.<DATE>T03:10:33.962Z info vpxd[<06#00>] [Originator@6876 sub=InvtVm opID=<HB...4b50>] Matched discovered VM (ds:///vmfs/volumes/<65...ca30>/<JG...VC08>/<JG...VC08>.vmx, vpxaVmid <18#18>) on host [vim.HostSystem:<ho...4127>,<jg...rnal>] with existing VM: [vim.VirtualMachine:<vm...8647>,<JG...VC02>] on host [vim.HostSystem:<ho...8215>,<pr...rnal>]<DATE>T03:10:34.711Z error vpxd[<07#78>] [Originator@6876 sub=drsExec opID=<Cd...r-01>] Failed migrating VM [vim.VirtualMachine:<vm...8647>,<JG...VC02>] to host vim.HostSystem:<ho...4127><DATE>T03:10:34.712Z error vpxd[<07#78>] [Originator@6876 sub=Default opID=<Cd...r-01>] [VpxLRO] -- ERROR task-<54...6679> -- -- <JG...VC02> -- Drm.ExecuteVMotionLRO: :vmodl.fault.HostCommunication<DATE>T03:16:47.382Z info vpxd[<06#79>] [Originator@6876 sub=licenseServiceDiscovery opID=<q-...10-38>] Discovered license service endpoint URL: https://<JG...rnal>:443/ls/sdk<DATE>T04:43:46.310Z info vpxd[<06#85>] [Originator@6876 sub=LogBundler opID=<mk...85-d0>] Directory: /etc/vmware-vpx/docRoot/diagnostics, bundle file name: /etc/vmware-vpx/bundle-stage/vc-<JG...rnal>-<DATE>--04.37-4108868.tgz, new file name: /etc/vmware-vpx/docRoot/diagnostics/vcsupport-<52...879b>.tgz - Running tasks appear frozen or remain stuck in a persistent "Busy" state within the vSphere Client UI.
- Thread scheduling saturation info:
tail -n 5000 /var/log/vmware/vpxd/vpxd.log | grep -o 'sub=[^ ]*' | sort | uniq -c | sort -nr3619 sub=vpxLro144 sub=StatsRegistry105 sub=MoCluster63 sub=drmLogger
Environment
VMware vSphere ESXi
VMware vSphere vCEnter Server
Cause
This issue is caused by thread starvation within the vCenter Server Management Daemon (vpxd) parent task scheduler (vpxLro pool).
When any background service loop, un-throttled security sync, or stale tracking task floods the scheduling queue, it exhausts global worker thread allocations.
This prevents vCenter from processing routine cluster heartbeats during high-load operations, resulting in transient host connection drops and migration timeout faults.
Resolution
Remediation depends on the system's ability to clear the transaction backlog:
- Automated Self-Recovery:
- If the management plane automatically regains responsiveness immediately following the completion of the cluster event (e.g., as seen above where the logs confirm the
VMOTION_RECOVERstate steal routine successfully cleared the transaction queues and system communication stabilized), no further action is required.
- If the management plane automatically regains responsiveness immediately following the completion of the cluster event (e.g., as seen above where the logs confirm the
- Manual Intervention: If specific hosts or tasks remain locked in a permanent "Busy" state, execute the following steps sequentially:
- Log into the vCenter Server Appliance as
rootand recycle the core service daemon to flush the blocked transaction queues from memory:service-control --stop vmware-vpxd && service-control --start vmware-vpxd- Note: This action restarting vpxd will stop tasks like backup, replication etc.
- Note: This action restarting vpxd will stop tasks like backup, replication etc.
- Log into the vCenter Server Appliance as
Additional Information
A similar background service loop or thread saturation mechanism can also impact host maintenance operations, causing related workflows to stay stuck in a stale or busy state.
Feedback
