Stale TKGi/PKS Logical Routers and Objects Cleanup in NSX-T

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware NSX

Issue/Introduction

Orphaned logical routers and stale objects remain in NSX-T after a non-graceful removal of the Tanzu Kubernetes Grid Integrated (TKGi) Control Plane. These stale objects continue to consume IP addresses from IP Pools, leading to IP pool exhaustion, "BGP down" errors, and "EDGE exceeded" deployment failures. This article provides the procedure to identify and clean up these orphaned resources using the nsx-cluster-cleanup.py script.

Environment

• VMware NSX-T (All versions)
• VMware Tanzu Kubernetes Grid Integrated Edition (TKGi) (All versions)

Cause

TKGI logical routers and associated objects remain in NSX-T if the TKGi Control Plane VMs are removed non-gracefully without running standard cleanup procedures. These stale objects continue to consume IP addresses from IP Pools, leading to resource exhaustion.

Resolution

Warning: Ensure you identify the correct cluster UUID before running the deletion script to avoid impacting production services.

1. Identify the stale TKGI cluster UUID:
   
   
   • Navigate to NSX Manager UI > Networking > Tier 1 Gateways.
   • Locate the routers with the naming pattern pks-****-cluster-router. Where **** is the cluster UUID
   • Copy the associated Cluster UUID from the object tags.
2. Download the nsx-cluster-cleanup.py script attached to this KB.
3. Execute the script using a user account with the enterprise_admin role in NSX Manager.
4. Run the script in list mode to verify the resources:
   

   python3 nsx-cluster-cleanup.py list --nsx-host <nsx host ip> --instance-id <cluster UUID> --api-type <policy/mp> --client-cert-path <nsx.pem> --client-key-path <nsx.key> --nsx-ca-cert-path <nsx-ca.pem>
   
   

5. After confirming the objects correlate to the identified cluster UUID, run the script in delete mode:
   

   python3 nsx-cluster-cleanup.py delete --nsx-host <nsx host ip> --instance-id <cluster UUID> --api-type <policy/mp> --client-cert-path <nsx.pem> --client-key-path <nsx.key> --nsx-ca-cert-path <nsx-ca.pem> --yes
   
   

NOTE:

• If you do not have certificate-based authentication, substitute --client-cert-path and --client-key-path with --username and --password parameters.
• If running from NSX manager itself, skip SSL validation with the --insecure flag
  
  Example:
  
  
  • List resources:

python3 nsx-cluster-cleanup.py list --nsx-host <nsx host ip> --instance-id <cluster-uuid>  --insecure --username <username> --password '<password>'
    
    
  • Delete Resources:
    
    python3 nsx-cluster-cleanup.py delete --nsx-host <nsx host ip> --instance-id <cluster-uuid> --insecure --username <username> --password '<password>' --yes

Additional Information

• If issues persist after running the cleanup, Contact Support for assistance.
• Refer to your specific NSX-T documentation for managing Tier-1 Gateways and IP Pool usage.
• Refer to Persistent "NCP plugin down" Alarm in NSX UI Due to Stale TKGI/WCP Cluster Entries for WCP cluster cleanup.
• If the TKGI Control Plane is still present, reference How to clean up NSX Protected objects created by TKGI for TKGI cluster cleanup.