Removing Tanzu Supervisor Cluster stalls during NSX resource cleanup
search cancel

Removing Tanzu Supervisor Cluster stalls during NSX resource cleanup

book

Article ID: 368072

calendar_today

Updated On:

Products

VMware vSphere with Tanzu

Issue/Introduction

Unable to remove Tanzu Kubernetes Cluster successfully. During the removal process it hangs indefinitely on 'NSX resource cleanup is in process'

Cause

NSX policies (gateways, groups, load balancer pools, segments, etc.) tied to the cluster domain when the Tanzu Kubernetes Cluster was created are still in use by NSX

 

Resolution

Find the NSX policy still in use from the vCenter wcpsvc.log. This log file is located under /var/log/vmware/wcp

This will be indicated by 'deleted' showing > 0. Followed by an error attempting to remove

2024-05-16T20:01:59.157Z info wcp [kubelifecycle/cluster_network.go:197] [opID=6 644029a-880867e7-918b-4fb0-b899-d27368203373] NSX cleanup for cluster domain-c8: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx stdout Using NSX manager endpoint 1: IP_ADDRESS:443

NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/domains
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-0s/domain-c8:CLUSTER_GUID
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/domain-c8:CLUSTER_GUID
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/domains/domain-c8:CLUSTER_GUID/security-policies
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/domains/default/security-policies
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/domains/domain-c8:CLUSTER_GUID/gateway-policies
Number of security policies to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/domains/mgw/gateway-policies/internal/rules
Number of internal gateway security policy rules to be deleted in domain mgw: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-0s/vmc/locale-services
Error: No locale services found, skip cleaning T0 Route Aggregation entries
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/services
Number of services to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/domains/domain-c8:CLUSTER_GUID/groups
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/domains/default/groups
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/domains/mgw/groups
Number of Groups to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/search?query=resource_type:IpAddressAllocation AND tags.scope:ncp\/cluster AND tags.tag:domain\-c8\:
Number of NCP IP allocations to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/lb-virtual-servers
Number of loadbalancer virtual servers to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/lb-services
Number of Loadbalance services to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/lb-pools
Number of loadbalancer pools to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/lb-persistence-profiles
Number of persistence profiles rules to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/search?query=resource_type:SegmentPort AND tags.s cope:ncp\/cluster AND tags.tag:domain\-c8\:
Number of NCP segment ports to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/search?query=resource_type:SegmentSecurityProfile BindingMap AND tags.scope:ncp\/cluster AND tags.tag:domain\-c8\:
Number of SegmentSecurityProfileBindingMap to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/domain-c8:CLUSTER_GUID/nat/INTERNAL/nat-rules
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/domain-c8:CLUSTER_GUID/nat/USER/nat-rules
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/domain-c8:CLUSTER_GUID/nat/DEFAULT/nat-rules
Number of nat rules to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/segments
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/domain-c8:CLUSTER_GUID/segments
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/ext-shared-services/segments
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/ext-tenant-mission-systems/segments
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/ext-vdi-instances/segments
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/int-shared-services/segments
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/int-tenant-mission-systems/segments
Number of Segments to be deleted: 0
Number of Segments to be disconnected: 0
Number of Avi SE segment ports to be deleted: 0
Number of Avi Segments to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/search?query=resource_type:IpAddressPoolBlockSubn et AND tags.scope:ncp\/cluster AND tags.tag:domain\-c8\:
Number of IpAddressPoolBlockSubnet to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/search?query=resource_type:IpAddressPoolStaticSub net AND tags.scope:ncp\/cluster AND tags.tag:domain\-c8\:
Number of IpAddressPoolStaticSubnet to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/ip-pools
Number of IpAddressPool to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s
Number of Tier-1s to be deleted: 1
Skip deleting the top tier1 router domain-c8:
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-0s
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-0s/tm-tier-0-gateway/static-routes
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-1s/domain-c8:CLUSTER_GUID/static-routes
Number of static routes to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/ip-pools
Number of IP Pools to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/certificates
Number of l7 resource certs to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/spoofguard-profiles
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/segment-security-profiles
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/ip-discovery-profiles
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/mac-discovery-profiles
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/qos-profiles
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/port-mirroring-profiles
Number of switching profiles to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/lb-app-profiles
Number of application profiles to be deleted: 0
NSXMGR_IP_ADDRESS:443/api/v1/systemhealth/container-cluster//ncp/status
Container system health status not found: 
NSXMGR_IP_ADDRESS:443/api/v1/fabric/container-clusters/
Inventory resource not found: 
Found Tier0 id tm-tier-0-gateway
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/tier-0s/tm-tier-0-gateway/route-maps
Number of Tier0 RouteMap to be modified/deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/search?query=resource_type:PrefixList AND tags.sc ope:ncp\/cluster AND tags.tag:domain\-c8\:
Number of Tier0 PrefixList to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/dhcp-server-configs
Number of DHCP server configs to be deleted: 1
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/dhcp-server-configs/dhcp_conf_domain-c8:CLUSTER_GUID
Exception {
  "httpStatus" : "BAD_REQUEST",
  "error_code" : 500030,
  "module_name" : "Policy",
  "error_message" : "The object path=[/infra/dhcp-server-configs/dhcp_conf_domai n-c8:CLUSTER_GUID] cannot be deleted as either it has children or it is being referenced by other objects path=[/infra/segments/nsx-seg-vdi-clients]"
} occurred, retrying after 60 seconds
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/dhcp-server-configs/dhcp_conf_domain-c8:CLUSTER_GUID
Exception {
  "httpStatus" : "BAD_REQUEST",
  "error_code" : 500030,
  "module_name" : "Policy",
  "error_message" : "The object path=[/infra/dhcp-server-configs/dhcp_conf_domai n-c8:CLUSTER_GUID] cannot be deleted as either it has children or it is being referenced by other objects path=[/infra/segments/nsx-seg-vdi-clients]"
} occurred, retrying after 60 seconds
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/dhcp-server-configs/dhcp_conf_domain-c8:CLUSTER_GUID
Exception {
  "httpStatus" : "BAD_REQUEST",
  "error_code" : 500030,
  "module_name" : "Policy",
  "error_message" : "The object path=[/infra/dhcp-server-configs/dhcp_conf_domai n-c8:CLUSTER_GUID] cannot be deleted as either it has children or it is being referenced by other objects path=[/infra/segments/nsx-seg-vdi-clients]"
} occurred, retrying after 60 seconds
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/dhcp-server-configs/dhcp_conf_domain-c8:CLUSTER_GUID
Maximum number of retries 3 exceeded, stopping iteration for function delete_policy_resource_by_path
NSXMGR_IP_ADDRESS:443/policy/api/v1/aaa/object-permissions
Number of node segment ports' hyperbus to be cleaned: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/search?query=resource_type:Ipv6NdraProfile AND ta gs.scope:ncp\/cluster AND tags.tag:domain\-c8\:
Number of ipv6 ndra profile to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/ip-blocks
Number of ip blocks to be deleted: 0
NSXMGR_IP_ADDRESS:443/policy/api/v1/infra/ip-pools
Number of external IP Pools to be deleted: 0
ERROR: Cleanup failed! Please try again.
ERROR: Failed to delete DHCP server config dhcp_conf_domain-c8:CLUSTER_GUID, error {
  "httpStatus" : "BAD_REQUEST",
  "error_code" : 500030,
  "module_name" : "Policy",
  "error_message" : "The object path=[/infra/dhcp-server-configs/dhcp_conf_domai n-c8:CLUSTER_GUID] cannot be deleted as either it has children or it is being referenced by other objects path=[/infra/segments/nsx-seg-vdi-clients]"
}
:
stderr:

 

After identifying the NSX policy still in use remove the associated object identified from within the NSX Manager UI. 

Powered by Wolken Software