Records of detached NSX manager nodes exist after deactivating the cluster

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You're running NSX 3.2.2
• You've recently deactivated the management cluster in NSX
• Running the following command on ESXi host shows manager nodes that no longer exist:

esx-01> nsxcli -c get controllers
<time-stamp>
 Controller IP Port SSL Status Is Physical Master Session State Controller FQDN
 <removed-nsx-mgr-ip> 1235 enabled not used false null NA
 <esxiting-nsx-mgr-ip> 1235 enabled connected true up NA
 <removed-nsx-mgr-ip> 1235 enabled not used false null NA

• Result of the follow API call shows controllers that no longer exist in the environment:

GET /api/v1/cluster/nodes

...
{
  "controller_role" : {
    "control_plane_listen_addr" : {
      "ip_address" : "<removed-nsx-mgr-ip>",
      "port" : 1235,
      "certificate" : "-----BEGIN CERTIFICATE-----
...
"resource_type" : "ClusterNodeConfig",
  "id" : "<removed-nx-mgr-uuid>",
...
  "_revision" : 2
}
...

• If you run the following command on the NSX manager admin CLI you observe only one node:

nsx-mgr-01.corp.local> get cluster config
<time-stamp>
Cluster Id: <cluster-uuid>
Cluster Configuration Version: 6
Number of nodes in the cluster: 1
Node UUID: <esxiting-nsx-mgr-ip>
Node Status: JOINED
 ...

Environment

VMware NSX-T Data Center

Resolution

This is a known issue impacting VMware NSX.

Workaround:
Make sure you have an up to date backup in place before proceeding.
Please run the following API calls to remove the records of the controllers that are no longer in use:

• POST /api/v1/cluster/nodes/deployments/<removed-nsx-mgr-uuid>?action=delete&force_delete=true
• POST /nsxapi/api/v1/cluster?action=deregister_local_mpa

{"node_uuid": <removed-nsx-mgr-uuid>}

• DELETE /nsxapi/api/v1/cluster/nodes/<removed-nsx-mgr-uuid>