Stale Antrea Cluster in NSX

Products

• VMware NSX

Issue/Introduction

A stale Antrea cluster name is visible in NSX Traceflow UI. 

The reported Antrea Cluster no longer exists in the customer environment. 

The following API, however, still references the non-existent Antrea cluster: 

curl -k -u 'admin:ADMIN_PASSWORD' https://NSX_MANAGER_IP/api/v1/infra/sites/default/enforcement-points/default/cluster-control-planes


{
  "results" : [ {
    "node_id" : "####-####-####-####",
    "resource_type" : "ClusterControlPlane",
    "id" : "###",
    "display_name" : "###",
    "path" : "/infra/sites/default/enforcement-points/default/cluster-control-planes/###",
    "relative_path" : "###",
...
}

Environment

VMware NSX

Cause

Stale data can occur when the cluster was registered manually (thus requiring a manual deregistration), and then the K8s cluster was deleted without running the deregistration job. 

Reference Doc

Since vSphere 8.0.3, the product provides an auto-registered + auto-cleanup workflow; this kind of issue won't happen in the automated workflow.

Resolution

The following command can be used to delete that cluster control plane resource.

1. Verify & delete guest TKC from NSX Inventory via APIs if exist

curl -k -n -X GET https://NSX_MANAGER_IP/api/v1/fabric/container-clusters/
curl -k -n -X DELETE https://NSX_MANAGER_IP/api/v1/fabric/container-clusters/<stale-cluster-id>

2. Delete Cluster enforcement end-points

curl -k -u admin -X DELETE -H "X-Allow-Overwrite: true" https://NSX_MANAGER_IP/policy/api/v1/infra/sites/default/enforcement-points/default/cluster-control-planes/###?cascade=true

3. Restart interworking
kubectl scale deployment interworking --replicas=0 -n vmware-system-antrea
kubectl scale deployment interworking --replicas=1 -n vmware-system-antrea

Additional Information

The customer may also receive "Control Channel To Antrea Cluster Down Long " alarm raised on the NSX system due to the stale Antrea cluster entry. 

https://knowledge.broadcom.com/external/article?articleNumber=409210