Updating outdated Ip pools to resolve control plane connectivity inaccessibility issues

Products

• VMware Tanzu Kubernetes Grid Management
• VMware Avi Load Balancer

Issue/Introduction

• Administrators are unable to run kubectl commands against a Tanzu workload cluster, receiving TLS verification anomalies and timeouts.
  

  kubectl get pods -A
  Unable  to connect to the server <vip-ip-address>: connect: no route to host

• A control plane node becomes trapped in a continuous recreation loop, causing etcd quorum to drop to less members.
• Tanzu management cluster inaccessible after control plane node rotation
• Avi load balancer virtual service is red because its backend pool continues to reference old, defunct control plane node IPs instead of the newly assigned IPs. 
• Node movements due to ESXi host patch update might be the trigger of ip misalignment
• Restart of ako pod does not result in getting the new ips updated

Environment

TKGm: 2.3.1

AVI: 22.1.3

Cause

• The Avi load balancer backend pool becomes out of sync following an automated machine health check recreation cycle, leaving the Virtual Service pointing to defunct control plane IPs and blocking cluster ingress management.

Resolution

1. Log into the primary Avi Controller UI
2. Navigate to Applications > Virtual Services or Pools.
3. Locate and select the Virtual Service / backend pool mapped to the affected management cluster.
4. Click on the action menu and select Edit Pool.
5. Scroll down to the pool members section, locate the old, defunct control plane node IP addresses, and remove them from the configuration.
6. Add the newly provisioned control plane node IP addresses with the corresponding API server port 6443 into the active members list.
7. Save the configuration changes and verify that the Virtual Service health monitor status transitions to an UP (Green) state.
8. Validate cluster accessibility by running kubectl commands from jump node or cluster environment utilizing the management cluster kubeconfig.