Global Managers Become Unavailable During CBM-Corfu Certificate Rotation via CARR Script

Products

• VMware NSX

Issue/Introduction

• Execution of the CARR script to rotate CBM_CORFU certificates results in a cluster failure.

• Following are the log snippet from /var/log/carr/carr.log.
  

  <Timestamp> cp -p /config/cluster-manager/corfu/private/keystore.jks /config/cluster-manager/corfu/private/keystore.jks_backup_20260616-093219;
  <Timestamp> - carr.interface.cli.keytool_intf - MainThread - INFO - keytool_intf.py:154 - Running command keytool -delete -alias "self" -keystore /config/cluster-manager/corfu/private/keystore.jks -storepass `cat /config/cluster-manager/corfu/private/keystore.password`

• Global Managers become offline or unreachable, which is also verified from /var/log/cbm/cbm.log.


  <Timestamp>  WARN CorfuRuntime-0 CorfuRuntime 74971 fetchLayout: latest layout epoch 798 > received 797 from <Mgmt_node_IP>:9000, discarded.
  <Timestamp>  WARN netty-0 ClientResponseHandler 74971 Server threw exception for SERVER_ERROR with request_id: 408234617
  <Timestamp>  WARN ClusteringRpcServer-Leadership-Thread1 AbstractView 74971 Got a wrong epoch exception, updating epoch to 799 and invalidate view
  <Timestamp> INFO ClusteringRpcServer-Leadership-Thread1 AbstractView 74971 layoutHelper: Retried 13 times, SystemDownHandlerTriggerLimit = 90
  <Timestamp> ERROR HeartbeatServiceServiceMonitorStatusUpdaterThread ServiceMonitor 74971 - [nsx@6876 comp="global-manager" errorCode="HBS153" level="ERROR" s2comp="service-monitor" subcomp="cbm"] One or more services are down: [Epoch:45]SEARCH:DOWN,GM:DOWN,AR:DOWN,HTTP:DOWN,SM:DOWN,MESSAGING_MANAGER:DOWN,PROTON:DOWN,CM_INV:DOWN,CLUSTER_MANAGER:UP,MONITORING:DOWN
  <Timestamp>  WARN ClusteringRpcServer-Heartbeat-Thread1 MultiVersionObject 74971 SnapshotProxy[2f54] encountered an exception during sync to 2152211421 on attempt 1 of 2
  com.vmware.nsx.platform.clustering.persistence.exceptions.CorfuShutdownException: Disconnected from database. Terminating thread.
          at com.vmware.nsx.cbm.factory.CorfuSystemDownHandler.run(CorfuSystemDownHandler.java:12) ~[libcbm.jar:?]
          at org.corfudb.runtime.view.AbstractView.layoutHelper(AbstractView.java:176) ~[runtime-4.2.20250423003417.9638.1.jar:?]
          at org.corfudb.runtime.view.AbstractView.layoutHelper(AbstractView.java:61) ~[runtime-4.2.20250423003417.9638.1.jar:?]
  <Timestamp> ERROR ClusteringRpcServer-Heartbeat-Thread1 MVOCorfuCompileProxy 74971 Access[ImmutableCorfuTable[2f54]] com.vmware.nsx.platform.clustering.persistence.exceptions.CorfuShutdownException: Disconnected from database. Terminating thread.
  ....
  Caused by: com.vmware.nsx.platform.clustering.persistence.exceptions.CorfuShutdownException: Disconnected from database. Terminating thread.

Environment

VMware NSX

Cause

This is caused by two compounding issues during the certificate rotation process:

1. Certificate Stream Listener Failure: During the operation, the CBM certificate stream listener becomes unsubscribed and fails to successfully re-register. This stems from underlying services improperly handling the SystemDownHandler when a node undergoes a service restart.

2. Incomplete Truststore Synchronization: While the CARR script successfully renews the CBM_CORFU certificate and applies it to both the Corfu keystore and the client truststore, it fails to update the Corfu server's truststore. This results in a cryptographic mismatch; nodes begin presenting a new certificate that peer nodes do not yet trust. This strictly isolates the servers, breaks cluster communication, and brings down the Global Managers.

Resolution

Note:  If your Global Managers are currently offline due to the certificate rotation issue, manual intervention is required to restore communication between nodes.

• Issue 1: Stream Listener Failure This is a known software defect. A permanent fix is included in NSX 9.0 and later. 

• Issue 2: CARR Script Truststore Gap A known issue prevents new certificates from being successfully applied to all nodes during the rotation process. A permanent script update is actively in development.

Workaround: Restore the NSX manager using NSX backup.  Please contact Broadcom Technical Support immediately for recovery assistance.