Removing stale WCP cluster that are causing a false positive report on NCP down alarm

Products

VMware NSX Networking VMware NSX

Issue/Introduction

NCP plugin down alarm present in NSX UI and not able to resolve the alarm as the alarm reappears

One or more clusters show as unknown in the output of the NSX API GET /api/v1/systemhealth/container-cluster/ncp/status:

{
"results" : [ {
"cluster_id" : "########-####-####-####-############,
"cluster_name" : "domain-##:#########-####-####-####-############",
"type" : "Kubernetes",
"status" : "UNKNOWN", <<<<<<<<<<<<<
"detail" : "",
"_protection" : "NOT_PROTECTED"
} ],
"result_count" : 2,

The POD health and manager connectivity is good by following the steps in Alarm for NCP health.
The Inventory > Containers > Clusters page in the NSX UI shows one or more Supervisor clusters that no longer exist. Similar evidence of these clusters will likely exist under Networking > Load Balancing as well.
From the VC support bundle, wcp-db-dump.py.txt will show only the healthy cluster defined. Make sure there is no entry found for the unknown cluster(s).
NSX support bundle reveals many resources still belonging to the unhealthy cluster "domain-##:#########-####-####-####-############", however you are not able to find this unhealthy cluster present in vCenter.

Environment

VMware NSX-T Data Center
VMware NSX
VMware vCenter Server
vSphere Supervisor services deployed/removed

Cause

NSX manager reports cluster as unhealthy due to stale inventory entries in NSX for a deleted WCP cluster and that is causing a false positive in reporting the NCP alarm down.
An NSX cleanup when the cluster was destroyed was not performed.

Resolution

The customer needs to cleanup resources for this WCP cluster using the below scripted approach by calling the unknown cluster name from API: GET /api/v1/systemhealth/container-cluster/ncp/status. This means removing all NSX resources that are referring to this cluster, including the inventory resources that are causing the alarm.

The script can be executed as follows.

Dry Run Mode - This will list the objects about to be cleaned up.
python3 /usr/lib/vmware-wcp/nsx_policy_cleanup.py --cluster <cluster id> -u <nsx admin user> -p '<nsx mgr admin pass>' --mgr-ip=<nsx mgr ip> --no-warning --top-tier-router-id=<iter0 id> --all-res

Actual Cleanup
python3 /usr/lib/vmware-wcp/nsx_policy_cleanup.py --cluster <cluster id> -u <nsx admin user> -p '<nsx mgr admin pass>' --mgr-ip=<nsx mgr ip> --no-warning --top-tier-router-id=<tier0 id> --all-res -r

Example:
python3 /usr/lib/vmware-wcp/nsx_policy_cleanup.py --cluster #########-####-####-####-############ -u <nsx admin user> -p '<nsx mgr admin pass>' --mgr-ip=<nsx mgr ip> --no-warning --top-tier-router-id=#########-####-####-####-############ --all-res -r

IMPORTANT: The -r option perform the actual removal of NSX resources. It is advised to first run the script without this option. This will perform a dry-run, users can evaluate the output to assess which resources will be removed before doing the actual deletion. Ensure a recent backup of NSX is available and that the backup passphrase is known.

Note: The <cluster id> value can be obtained via the NSX API GET /api/v1/systemhealth/container-cluster/ncp/status API call noted in the Issue section. Similarly, the <tier0 id> value can be obtained via NSX API GET /api/v1/infra/tier-0s.

Additional Information

If the vCenter Server is version 8.0.3:

This will require triggering the script in an alternate way, due to the presence of an envoy proxy in VCSA.

In this case as the cleanup operation will also need to go through the envoy proxy, users should set the following options:

--envoy-endpoint: IpAddress or Hostname of VC envoy sidecar, ignored if the environment variable ENVOY_ENDPOINT is set
--envoy-port: HTTP port of VC envoy sidecar, ignored environment variable ENVOY_PORT is set. Defaults to 1080.

Alternatively, users can copy the cleanup script to any machine with connectivity with NSX manager and do the cleanup from there; the script indeed has no dependency on WCPSVC. Users should just be aware that the script requires the python requests package which must be installed on the machine (or python venv) from which the script is executed.

Cleaning Up the NSX Environment