Supervisor Cluster Upgrade stuck at 15% with "Apply Solution" Health Check Failed Error

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server
• VMware NSX

Issue/Introduction

Upgrading a vSphere Supervisor cluster fails and gets stuck at 15% during the "Apply Solution" task. ESXi hosts fail to go into Maintenance Mode which stops spherelet from being upgraded.

In the vum-server.log, errors similar to the following are observed:

error vmware-vum-server[XXX] [Originator@6876 sub=ClusterApplySolutionTask] [Task, 524] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplySolutionTask ID:XXX. Task Failed. Error: Error:
-->    com.vmware.vapi.std.errors.error
--> Messages:
-->    com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed<Health Check for 'esx1' failed>

error vmware-vum-server[XXX] [Originator@6876 sub=EHP opID=XXX] Response from localhost/external-tp/http1/NSX_MGR/443/XXX/api/v1/vlcm/esx/health/cluster/perspectives/planned-maintenance/status?action=check: HTTP Status:500 'Internal Server Error'

In the wcpsvc.log, errors similar to the following are observed:

error wcp [pman/client.go:659] [opID=vLCM:Upgrade:esx1] PMan API: Task Get for Apply Task API: Attempt#[5]:Failed Attempts:[0] of MaxFailedAttemps[1]Error Message: Health Check for 'esx1' failed
error wcp [kubelifecycle/pman_client.go:488] [opID=vLCM:Upgrade:esx1] PMan API: ApplyUpgradeTask FAILED, PMan API: Apply Task: ***FAILED: Multiple retries completed

Environment

vCenter 8.0u3
Supervisor Cluster v1.29.x to v1.30.x
VMware NSX 4.2

Cause

The issue occurs because the NSX Upgrade Coordinator service is permanently stopped on the NSX Manager nodes following an extended period of connectivity timeouts (e.g., resulting from an earlier Corfu cluster disruption). This prevents vLCM health check API calls from reaching the Upgrade Coordinator, returning an HTTP 500 upstream failure. The failure of these health checks causes the Supervisor "Apply Solution" task to halt and fail.

Resolution

1. Log into each NSX Manager node via SSH using admin credentials.

2. Execute the following command to restart the Upgrade Coordinator service: /etc/init.d/upgrade-coordinator restart

3. Verify that the service is up and responding cleanly by executing: curl -k https://localhost/api/v1/upgrade/status-summary

4. Once the service is active on all NSX Manager nodes, the Envoy proxy will successfully route vLCM health check API requests. The pending vLCM "Apply Solution" tasks will then be able to complete normally.