NSX Global Manager Upgrade Paused Due to Corfu Quorum Timeout

Products

• VMware NSX

Issue/Introduction

• During the upgrade of an NSX Global Manager (GM) cluster, the upgrade process enters a "Paused" state following the reboot of an individual manager node.
• If the upgrade workflow is started again, the status goes back into "Paused" state after a few minutes.
• Following error may be observed in /var/log/corfu/corfu.9000.log on the Manager nodes:

| ERROR |              DetectionWorker-1 |  o.c.i.RemoteMonitoringService | Failure detection task finished with an error
java.util.concurrent.CompletionException: org.corfudb.runtime.exceptions.unrecoverable.UnrecoverableCorfuInterruptedError: java.lang.InterruptedException: sleep interrupted

• Following error may be observed in /var/log/upgrade-coordinator/upgrade-coordinator log

ERROR Thread-9 MPClusterUpgradeServiceImpl 2534824 SYSTEM [nsx@4413 comp="nsx-manager" errorCode="MP30447" level="ERROR" logger="MPClusterUpgradeServiceImpl" msgID="SYSTEM" subcomp="upgrade-coordinator" threadName="Thread-9"] Exception invoking playbook.\ncom.vmware.nsx.management.upgrade.rpcframework.UcRestRpcException: [UC] Error in rest call. url= /api/v1/cluster/########-a055-1df8-2d23-############/node/upgrade/performtask?action=execute , method= POST , response= {"error_code": 36580, "error_message": "Error proxying request to: ########-a055-1df8-2d23-############.", "module_name": "node-services"} , error= 500 Internal Server Error on POST request for "http://localhost:7441/api/v1/cluster/########-a055-1df8-2d23-############/node/upgrade/performtask": "{"error_code": 36580, "error_message": "Error proxying request to: ########-a055-1df8-2d23-############.", "module_name": "node-services"}" .\n

Environment

VMware NSX

Cause

A race timing condition within the clustering algorithm is exacerbated by high network latency during the rolling reboot process of the manager nodes. This prevents the remaining nodes from establishing a Corfu quorum within the defined database connection timeout period (by default, 120-seconds), causing the Upgrade Coordinator (UC) to trigger service restarts and pause the upgrade workflow.

Resolution

To resolve the paused state and allow the upgrade workflow to resume, perform the following steps on the NSX Global Manager nodes.

Prerequisites:

• Ensure console or SSH access is available for all Global Manager nodes.
• Verify the current status of the cluster nodes.

Steps:

1. Log in to the NSX Global Manager node as the root user.
2. Restart the corfu-server service on the first affected node:
   bash

    

   /etc/init.d/corfu-server restart

3. Monitor the status until the service is active:
   bash

    

   /etc/init.d/corfu-server status

4. Verify that the cluster returns to a "STABLE" state:
   bash

    

   get cluster status

   Alternatively, verify this in the UI under System > Configuration > Global Manager Appliances.
5. Repeat steps 1-4 for remaining Global Manager nodes as necessary.
6. Once all nodes report a stable status, restart the upgrade workflow from the Upgrade Coordinator UI.

 

If the issue persists after restarting the corfu-server service, open a Broadcom support case and refer to this KB article: Creating and managing Broadcom cases

Additional Information

The broader symptoms of a Corfu server QuorumUnreachableException and slow layout updates during a node failure or restart can also occur outside of upgrade scenarios. If the Corfu layout fails to stabilize after all nodes are back online, and there is no obvious infrastructure instability, we recommend performing a rolling restart of all Corfu servers.