ESXi Upgrade Fails During the REMEDIATE Stage via Fleet Manager Due to Active vSAN Health Alarms

Products

• VCF Operations
• VMware Cloud Foundation

Issue/Introduction

• An ESXi upgrade from VMware Cloud Foundation (VCF) 9.0 to 9.1 via VCF Fleet Manager fails during the REMEDIATE stage.
• Remediation failure is reported within the VCF Fleet Manager UI and upgrade logs due to active vSAN health alarms.
• Additionally, token validation failures can halt the REMEDIATE stage if clock skew exists between SDDC Manager and vCenter Server.

 

 

 

Environment

VMware Cloud Foundation 9.x

Cause

• The upgrade is blocked by strict vSAN health checks enforced by VCF Fleet Manager during the REMEDIATE phase, triggered by underlying NVMe device and SCSI controller warnings.
• Host remediation is prevented by Fleet Manager if vSAN health alarms are active to protect cluster data integrity.

Resolution

• Silence the active vSAN health alarms (NVMe devices and SCSI controllers) in the vSphere Client to bypass the Fleet Manager health check.
• Retry the host remediation via VCF Fleet Manager.
• If specific hosts continue to fail via Fleet Manager despite silenced alarms, upgrade the host manually using the interactive method via out-of-band management.
• Following the manual upgrade, re-run the upgrade configure task from VCF Fleet Manager to synchronize the SDDC Manager inventory.
• Post-upgrade, investigate and resolve the underlying NVMe and SCSI controller hardware faults.

Additional Information

Verification of NTP synchronization is recommended prior to remediation to ensure no clock skew exists between SDDC Manager and vCenter Server.