"Control Channel To Antrea Cluster Down Long " alarm raised on NSX system despite the antrea cluster does not exist

Products

• VMware NSX

Issue/Introduction

NSX manager reported the Control Channel to the Antrea Cluster Down long alarm:

 

However, the reported Antrea Cluster no longer exists in the customer environment. 

The following API, however, still references the non-existing Antrea cluster: 

curl -k -u 'admin:ADMIN_PASSWORD' https://NSX_MANAGER_IP/policy/api/v1/infra/sites/default/enforcement-points/default/cluster-control-planes


{
  "results" : [ {
    "node_id" : "####-####-####-####",
    "resource_type" : "ClusterControlPlane",
    "id" : "tkc-02",
    "display_name" : "###",
    "path" : "/infra/sites/default/enforcement-points/default/cluster-control-planes/###",
    "relative_path" : "###",
...
}

Environment

VMware NSX

Cause

Stale data can occur when the cluster was registered manually (thus requiring a manual deregistration), and then the K8s cluster was deleted without running the deregistration job. 

Reference Doc

Since vSphere 8.0.3, the product provides an auto-registered + auto-cleanup workflow; this kind of issue won't happen in the automated workflow.

Resolution

The following command can be used to delete that cluster control plane resource.

curl -k -u 'admin:ADMIN_PASSWORD' \
  -X DELETE -H "X-Allow-Overwrite: true" \
  https://NSX_MANAGER_IP/policy/api/v1/infra/sites/default/enforcement-points/default/cluster-control-planes/###?cascade=true