"The object cannot be deleted as either it has children or it is being referenced by other objects" when deleting a Load Balancer Virtual Service in Tanzu VKS

Products

• VMware NSX

Issue/Introduction

• A Load Balancer virtual service is created and subsequently deleted via VKS
• The virtual service IP is no longer visible in the cluster when running command:  'kubectl get service -A'
• When attempting to reuse this IP for another virtual service, the following errors are observed in the ncp logs

[ncp ] nsx_ujo.ncp.k8s.service_lb_controller Encountered retryable error while processing L4 Lb service ########-####-####-####-########: NSX IP ###.###.###.### is already allocated in ip pool ipp_########-####-####-####-########_ingress
[ncp ] nsx_ujo.common.controller ServiceLbController worker 2 failed to sync ########-####-####-####-######## due to retryable exception: NSX IP ###.###.###.### is already allocated in ip pool ipp_########-####-####-####-########_ingress
[ncp ] nsx_ujo.ncp.k8s.service_lb_controller Failed to process L4 Lb service ########-####-####-####-########: The object cannot be deleted as either it has children or it is being referenced by other objects

Environment

VMware NSX

VMware NSX-T Data Center

 

Cause

The NSX Container Plugin (NCP) attempts to delete the Virtual Service and its associated configurations (such as an NSGroup). This deletion fails at the NSX Manager API level if there are external dependencies linked to the object. For example, if an administrator manually added the NCP-created NSGroup to a custom Distributed Firewall (DFW) rule, NSX referential integrity prevents the deletion. The NCP deletion request eventually times out, resulting in a stale object remaining in the NSX inventory.

Resolution

This is a condition that may occur in a VMware NSX environment.

 

Workaround

To clear the stale object and allow NCP to synchronize:

1. Locate the stale NSGroup
Log in to the NSX Manager UI and navigate to Inventory > Groups. Use the search bar to find the NSGroup associated with the stale Load Balancer Virtual Service (typically identified by the Cluster ID or Service Name in the nsx-ncp logs). Click Members to verify it contains the relevant stale IP segments.

2. Clear Dependencies
Navigate to Security > Distributed Firewall (and Gateway Firewall). Search for the NSGroup identified in Step 1. If the group is used as a Source or Destination in any rules, you must delete those rules or remove the group from them. The object cannot be deleted while active dependencies exist.

3. Connect to the Cluster Context
Establish a connection to the Supervisor Cluster or Tanzu Kubernetes cluster using one of the following methods:

vSphere Plugin (Recommended): Run kubectl vsphere login --server=IP-OR-FQDN using your vCenter credentials.

Kubeconfig: If you have a direct kubeconfig file, export it: export KUBECONFIG=path/to/config.

Documentation Reference: Refer to the official VMware documentation: Connecting to a Tanzu Kubernetes Cluster.

4. Force NCP Resync
Delete the nsx-ncp pod to trigger a reconciliation:

kubectl delete pod <nsx-ncp-pod-name> -n vmware-system-nsx

Note: Deleting the pod is safe. The Deployment controller will automatically recreate and restart a new pod. Upon startup, the new pod performs a full state comparison between the Kubernetes API and NSX, which successfully clears the unreferenced stale objects.