Local NSX-T Managers nodes are showing in an "Unknown State" from the Global Manager UI after being replaced.

Products

• VMware NSX

Issue/Introduction

• You have recently tried to deactivate/detach the NSX Manager cluster.
• You have recently replaced one or more of your Local Manager nodes.
• Local Managers (LM) show Unknown State from Global Manager UI.
• All services on Global Manager show as stable.
• There is no functional impact of the Local Managers.
• Entries similar to the below will be seen on the Local NSX Manager due to the cluster deactivation failure:
  

  /var/log/nsx-audit.log
  2023-03-31T11:57:33.744Z nsxmgr-01 NSX 17233 - [nsx@6876 comp="nsx-manager" subcomp="cli" username="admin" level="INFO" audit="true"] CMD: detach node ########-####-####-####-############ (duration: 238.572s), Operation status: CMD_EXECUTED_WITH_ERROR_RESULT
  
  var/log/nvpapi/api_access.log
  2023-03-31T12:39:21.727Z INFO admin 'POST /api/v1/cluster-manager?action=deactivate_cluster' 500 414 "" "" 380.904066
  
  var/log/nvpapi/api_server.log
  2023-03-31T12:33:00.824Z napi.root.node.cluster INFO DeactivateCluster called.
  2023-03-31T12:39:21.726Z napi.root.cluster.cbm_response ERROR Error getting status for DeactivateCluster request, error_code 0, additional_info: [CBM238] Failed to deregister MPA with MP. Please make sure MP is stable, before retrying.

 

• In the Local NSX Manager support bundle, the desired_state_manager.json may contain entries you indicating that the control cluster status is in an unknown state.
  

  "node_status": {
  
    "control_cluster_status": {
  
      "control_cluster_status": "UNKNOWN",
  
      "mgmt_connection_status": {
  
        "connectivity_status": "CONNECTED"
  
  <SNIP for readability>
  
      "host_msg_client_info": {
  
        "account_name": "cvn-ccp-e040b066-xxxx-xxxx-xxxx-900a6bfadabc" 
  
      },
  
      "mpa_msg_client_info": {
  
        "account_name": "cvn-mp-mpa-9ca4a62b-xxxx-xxxx-xxxx-f02bdfe441f6"
  
      },
  
      "type": "ControllerClusterRoleConfig"
  
    }
  
  },

Environment

VMware NSX

Cause

The cause of this issue is that the Management Plane is not stable when the Deactivate Cluster operation is initiated, this results in stale entries in the Local NSX Managers.

Resolution

This is a condition that may occur in a VMware NSX environment.

Workaround

1. Collect and inspect the API output for stale NSX Manager nodes

   GET /api/v1/cluster/nodes/deployments

2. Delete the stale nodes.

   DELETE /api/v1/cluster/nodes/<node-UUID>