ESXi Host Reconnection Failure and EVC Mismatch During VCF/SDDC Manager Upgrade

Products

• VMware vSphere ESX 8.x

Issue/Introduction

 

• During a VMware Cloud Foundation (VCF) host remediation or upgrade workflow initiated via SDDC Manager, the ESXi host successfully completes the ESXi software installation but fails during the final remediation stage.

• SDDC Manager UI displays an LCM remediation failure with an error token stating:

  "VUM Remediation (installation) of an ESXi host failed. Manual intervention needed as upgrade failed during install stage."

• The ESXi host appears as disconnected in the vSphere Client.

• Attempting to manually reconnect the host in vSphere Client fails with the error:

  "Reconnect host failed! The host cannot be admitted to the cluster's current Enhanced vMotion Compatibility mode. Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode."

• The local ESXi Host Client UI shows an error indicating a VM failed to power off or migrate (e.g., Failed to power off virtual machine . The operation is not allowed in the current state).

 

Environment

VMware vSphere ESX 8.x

Cause

This issue is caused by a dual-condition blockage occurring during the host evacuation and maintenance mode exit phases of the VCF upgrade workflow, during host evacuation, a virtual machine fails to power off or migrate cleanly and enters a hung/suspended power-transition state.
While the active VM process terminates (esxcli vm process list returns empty), the VM object remains registered in the host inventory (vim-cmd vmsvc/getallvms).
When the host attempts to reconnect to vCenter while out of Maintenance Mode, vCenter's Enhanced vMotion Compatibility (EVC) engine inspects the host's active inventory. Because a registered VM is present exposing higher CPU feature masks, vCenter rejects the host admission attempt.

Resolution

To resolve the issue and allow SDDC Manager to complete the upgrade task:

Step 1: Clear the Stuck VM Registration via SSH
Connect to the affected ESXi host via SSH as root.

Check for active running VM processes:
esxcli vm process list
If no active processes are found, identify the stuck VM ID from the host inventory:

vim-cmd vmsvc/getallvms
Force power-off or unregister the stuck VM (e.g., VM ID-###):
vim-cmd vmsvc/power.off

If power.off fails, unregister the VM from the host:
vim-cmd vmsvc/unregister

Step 2: Enforce Maintenance Mode and Reconnect to vCenter
Place the host into Maintenance Mode via SSH to bypass EVC checks during reconnect:

esxcli system maintenanceMode set --enable true
Restart ESXi management services:
services.sh restart

Log into vSphere Client, right-click the disconnected host, and select Connection > Reconnect. 
Enter the host root credentials when prompted.

Step 3: Register the VM & Clear vSAN Delay Timers
Register the VM on the host or a healthy cluster node:
vim-cmd solo/registervms /vmfs/volumes///.vmx

Power on the VM and take the host out of Maintenance Mode:
esxcli system maintenanceMode set --enable false