NSX Transport Nodes Display Duplicate Entries in 'get controllers' Output Due to Stale Corfu Data

Products

• VMware NSX

Issue/Introduction

Transport Nodes experience connection refused errors to the NSX Manager over port 1235. Following the deactivation of the NSX Manager cluster and recovery of the NSX UI, transport nodes refuse to connect to the single NSX Manager node. Upon deploying two additional manager VMs and joining them to the cluster, transport nodes successfully connect but display duplicate entries in the get controllers output.

Diagnostic log signatures include:

• /etc/vmware/nsx/controller-info.xml on ESXi hosts shows duplicate <server> IP entries.

• nsx-syslog.log on ESXi Transport Nodes logs: ERROR: VersionMastershipHandshakeClient: received MasterResponse UUID <REDACTED_SECRETS> not in ....

• cbm.log on the NSX Manager logs: WARNING: Can't find entity corresponding to member <REDACTED_UUID> of group <REDACTED_UUID>.

Environment

VMware NSX

Cause

Decommissioned NSX Manager nodes were not successfully cleaned up from the cluster, leaving stale data in the ClusterNodeConfigModel Corfu table. The desired_state_manager.json file reads its content from this table. ControllerMgmtVertical queries the controller list from ClusterNodeConfigTable and sends this information to the host. The ops-agent on the host receives this data and writes it to controller-info.xml, resulting in duplicate entries.

Resolution

If you find that you have encountered this issue, please open a case with Broadcom Support and reference this KB: Creating and managing Broadcom cases

Additional Information

NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks Due to ctl Field Release Count (RC) Overflow