ESXi Host disconnected with error: "Disconnected from host: Reason: Agent is out of date and needs a manual upgrade"

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi host is up and accessible from web interface but showing disconnected in vCenter with message: "Disconnected from host: Reason: Agent is out of date and needs a manual upgrade"
  
  
  
  
• Manual reconnect tasks hang at 90% with the message: "Processing data from vCenter agent on host_name
• vCenter becomes unresponsive during host reconnection attempts, requiring vCenter vpxd service restart to bring it to the accessible state.
• /var/log/vmware/vpxd/vpxd-profiler.log on vCenter shows multiple threads stuck in WAITING_MOLOCK state for the affected Host_ID.
  
  ThreadState/ThreadId/####/State/Task::task-####::host-ID::vim.HostSystem.reconnect::#########/State/Vdb/State/Vdb
--> ThreadState/ThreadId/####/State/Task::lro-####::::VpxdInvtHostSyncHostLRO.Synchronize::
--> ThreadState/ThreadId/####/State/Task::lro-####::::VpxdInvtHostSyncHostLRO.Synchronize::
--> ThreadState/ThreadId/####/State/Task::lro-####::::VpxdInvtHostSyncHostLRO.Synchronize::
--> ThreadState/ThreadId/####/State/Task::lro-####::::VpxdInvtHostSyncHostLRO.Synchronize::
--> ThreadState/ThreadId/####/State/Task::lro-####::ResourceModel::vim.dp.ResourceModel.queryBatch::########/State/WAITING_MOLOCK -- host-ID::vim.HostSystem
--> ThreadState/ThreadId/####/State/Task::lro-####::::VpxdInvtHostSyncHostLRO.Synchronize::
#########/State/WAITING_MOLOCK -- host-ID::vim.HostSystem
--> ThreadState/ThreadId/####/State/Task::lro-####::::VpxdInvtHostSyncHostLRO.Synchronize::
################/State/WAITING_MOLOCK -- host-ID::vim.HostSystem

Environment

VMware vSphere ESXi 8.x

Cause

The issue is caused by a database level deadlock or a stale Managed Object lock (MOLOCK) in the vCenter Server Database (VCDB). This lock prevents vpxd from completing host synchronization.

Resolution

To resolve this issue, ESXi host must be removed from the vCenter Database to release the lock.

1. Log in directly to the affected ESXi Host UI and place the host into Maintenance Mode.
2. Go to vCenter, Right-click the affected ESXi host and select Remove from Inventory.
3. Connect to the vCenter Database (VCDB) and validate that the affected host is no longer present in VCDB:
   
   • Login to the vCenter via SSH using the root credentials
   • Connect to the vCenter Server vPostgres database:
     /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
     
     
   • Query the ESXi host in the vCenter database :
     select * from VPX_ENTITY where name like '%ESXi_FQDN%';
     
4. If the query returns no records, the host removal was successful. You can now add the ESXi host back to the vCenter inventory.
5. If the query returns the affected host record, the entry is stale and must be removed manually.
   
   • Take a offline snapshot of the vCenter appliance before proceeding.
   • Manually remove the host entries and clean up all metadata related to the affected host, following KB Manually removing an ESXi host from the vCenter Server database
6. Once the host entry is removed, add the ESXi host back to the vCenter inventory.
7. Exit host maintenance mode and validate host functionality.