Load balancer pool alarms "down status" requiring cleanup of obsolete pools

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware NSX for vSphere

Issue/Introduction

Customer receives "status down" alarms for Load Balancer pools in NSX-T which are not in use. The load balancer pools alarm because the backend services (LB targets) are no longer present. We need to verify that the targets are no longer present, and if so, then the LB pools can be deleted.

Environment

TKGi 1.20

Ops Manager 2.10.70

Resolution

To investigate the status of the LB pools and their components, and to clear the "down" status, do the following:

1. Go to NSX Manager, and find the Virtual Server that is the subject of the alarm.
   
   For example, here's a Virtual Server page:
   
   
   
   
2. Identify the TKGI cluster from the above page. The value of ncp/cluster shows the UUID of the TKGI cluster. Correlate this with the output of tkgi clusters to identify the right cluster.
3. Once the cluster is identified, identify the namespace within the cluster. The namespace is the Tag value of ncp/project.
4. Within the namespace, identify the service name. This is the value of k8s_resource_name.
5. Once the service is identified, confirm the status of the pods -- are they running or not? Run kubectl describe service $name to identify the pods behind the service.
6. Delete the service if it is no longer needed. For example, run kubectl delete service $name. After the service is deleted in the cluster, the NSX Virtual Server will be automatically deleted it is synced by the ncp job.