NSX Manager Cluster Upgrade Failed: Handover Timeout and Search Framework Errors

Products

VMware NSX

Issue/Introduction

During a cluster upgrade of the NSX Management Plane, the process fails on the final node in the cluster. While the initial nodes in the cluster upgrade successfully, the upgrade stops, and the UI reports the following error:

Unexpected error while upgrading upgrade unit: Failed to handover upgrade control to another node. Please ensure that the other nodes are functioning, and retry the upgrade.

Diagnostic logs from the upgrade-coordinator contain the following error: SEARCH_FRAMEWORK_INITIALIZATION_IN_PROGRESS:

"details" : "SEARCH_FRAMEWORK_INITIALIZATION_IN_PROGRESS",
"httpStatus" : "BAD_REQUEST",
"error_code" : 60526,
"module_name" : "nsx-search",
"error_message" : "Search framework is being initialized."
} , error= 400 : "{<EOL> "details" : "SEARCH_FRAMEWORK_INITIALIZATION_IN_PROGRESS",<EOL> "httpStatus" : "BAD_REQUEST",<EOL> "error_code" : 60526,<EOL> "module_name" : "nsx-search",<EOL> "error_message" : "Search framework is being initialized."<EOL>}" .
com.vmware.nsx.management.upgrade.rpcframework.UcRestRpcException: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : "{<EOL> "details" : "SEARCH_FRAMEWORK_INITIALIZATION_IN_PROGRESS",<EOL> "httpStatus" : "BAD_REQUEST",<EOL> "error_code" : 60526,<EOL> "module_name" : "nsx-search",<EOL> "error_message" : "Search framework is being initialized."<EOL>}"

This issue indicates that the nsx-search service is still re-indexing or initializing on the previously upgraded node, causing the Upgrade Coordinator (UC) API requests to time out or return 500 Internal Server Errors, resulting in a cluster desynchronization.

Environment

Product: VMware NSX (Management Plane)
Versions: Upgrade path including 4.2.x
Configuration: 3-node NSX Manager Cluster

Cause

The nsx-search service on the previously upgraded node did not complete its initialization phase before the upgrade sequence initiated for the subsequent node. This prevents the Upgrade Coordinator from querying essential transport node information, leading to a handover failure. A manual retry attempts to trigger an upgrade on an already desynchronized Management Plane, often resulting in further 500 Internal Server Errors due to inconsistent state across the cluster.

Resolution

Before proceeding, ensure you have a current backup/snapshot of the NSX Manager cluster.

Step 1: Validate Search Service Status

Log in to each NSX Manager node via SSH as admin.
Execute the following command to check the search service state:
```
get service search
```
If the service is not in the "Running" state, restart it:
```
restart service search
```
Wait 5 minutes and repeat step 2 to confirm the service is now "Running." Do not proceed until this service is stable on all nodes.

Step 2: Restart Upgrade Services

On each NSX Manager node via the admin CLI, execute the following commands to clear the stale "IN_PROGRESS" state:
```
restart service install-upgrade 
restart service upgrade-coordinator
```

Step 3: Verify Synchronization

Execute the following command to ensure all nodes report a consistent state:
```
get upgrade progress-status
```
Test API connectivity to the Search service to ensure it is responsive:
```
GET /api/v1/search/status
```

Step 4: Retry Upgrade Once all services display a "Running" state and the Search status returns "UP," retry the upgrade operation from the NSX UI.

If the cluster remains in a corrupted or desynchronized state: If the steps above fail to resolve the desynchronization, the cluster state may require a clean rebuild:

Roll back the Management Plane upgrade.
Remove one manager appliance at a time from the cluster.
Deploy new appliances and join them to the cluster sequentially.
Reattempt the upgrade.

Workaround

There is no temporary mitigation for this issue. The upgrade cannot proceed until the Management Plane service initialization and synchronization are verified.

Additional Information

Reference KB: KB 314346 Upgrade of an NSX cluster succeeds to upgrade only 2 nodes, failing to do a hand-over for the final node. Use this to verify that certificate attributes are not the cause of the handover failure before proceeding with search service troubleshooting).
For product defects or enhancement requests, subscribe to this article to receive automated updates.