NSX-T Cluster Upgrade Stuck Due to Control Plane Unknown State

Products

• VMware NSX

Issue/Introduction

During an NSX-T cluster upgrade, the upgrade process may stall or appear to be stuck. Symptoms include the control plane entering an "Unknown" state, and specific cluster nodes failing to progress.

This is often accompanied by continuous HTTP 400 INDEXING_FAILURES_EXHAUSTED_RETRIES (Error 60516) or INDEXING_IN_PROGRESS (Error 60508) errors returned from the NSX API to the NSX Container Plugin (NCP), which blocks load balancer updates and completely stalls Tanzu cluster upgrades.

Environment

VMware NSX

Cause

This issue is identified as a known defect  within the NSX Search framework. During a load balancer pool update, the search indexer may encounter a transient failure and hit its retry limit.

While the system's indexing process actually recovers and successfully indexes the objects in the background, a backend bug prevents the "exhausted retries" flag from resetting. Because this stale flag remains stuck in memory, the framework continuously throws the exhausted retries error back to NCP, causing the upgrade to stall even though the underlying data-plane processes are healthy.

Resolution

Perform the following steps to clear the stuck state

1. Log in to the NSX Manager appliance via SSH using the admin credentials and run the command  start search resync all. This forces the search framework to rebuild its index, which clears the stuck flag without needing a reboot.

2. Alternatively, if a CLI resync cannot be performed, execute a rolling reboot of all three NSX Managers to clear the memory state.

3. Restart the NCP (NSX Container Plugin) service on the affected nodes.

4. Monitor the status of the cluster upgrade from the NSX Manager UI to ensure progression resumes.

If the issue persists, please open a support request. To speak with a customer representative or a Support Engineer, see Contact Support. Scroll to the bottom of the page and click on your respective region.