How to recover supervisor and guest cluster control plane LB IP hosted on Avi Load balancer
search cancel

How to recover supervisor and guest cluster control plane LB IP hosted on Avi Load balancer

book

Article ID: 390201

calendar_today

Updated On:

Products

VMware Avi Load Balancer

Issue/Introduction

  • While trying to kubectl vsphere login to the Supervisor cluster or guest cluster, the below error is observed: 

    kubectl vsphere login --server=https://<current Workload Management Supervisor control plane endpoint>

ERRO[YYYY-MM-DD HH:MM:SS] Error occurred during HTTP request: Get "https://<current Workload Management Supervisor control plane endpoint>/wcp/loginbanner": tls: failed to verify certificate: x509: certificate is valid for <old Workload Management Supervisor control plane endpoint>, not <current Workload Management Supervisor control plane endpoint>
There was an error when trying to connect to the server.
Please check the server URL and try again.

     

  • IP addresses for Supervisor Cluster kube-apiserver, vsphere-csi-controller and Guest Cluster control plane services have changed.
  • External and internal communication to the Kubernetes API server (kube-apiserver) fails over TCP port 6443 across guest clusters. This is demonstrated by running kubectl commands from within the failing guest clusters.
  • Running kubectl get clusters -A on the supervisor shows guest clusters' AVAILABLE column with value False.

Environment

  • vSphere Supervisor v1.28 and lower
  • AVI-AKO

Cause

This is a known issue caused by the environment's AVI-AKO build version included with the vCenter and Supervisor cluster. 

Following scenarios can lead to a change in IP address of the control-plane services. 

  •  If kube-apiserver-lb-svc service gets impacted, it can lead AKO to delete the config on Avi. At a later point, when the service comes back up healthy, AKO will be notified to recreate the services on Avi. As all the virtual services get deleted and recreated by AKO, the IPs for virtual services (hosting control-plane svc) can change.

  •  In a SDDC-managed infra, if multiple vCenter Servers are sharing a single NSX cloud and a new cluster is created in a new workload domain, it can lead to the deletion of existing virtual services on Avi present for another cluster. This happens as the initials of the cluster name (domain-<>)  is auto-created and can end up being the same for two different clusters (part of different workload domain). This is currently an unsupported configuration, and the newly created cluster must be deleted followed by AKO restart on supervisor to recover the virtual services for the old cluster. 
    Reference document: Supervisor Networking

Resolution

This issue is resolved in vCenter 8.0u3E and Supervisor cluster version 1.29.7.

Workaround: The IP addresses for the services will need to be reverted to the original IPs, and virtual services deleted to be recreated within AVI.

Steps to restore IP in both VDS and NSX WCP deployments:

    1. Connect into the Supervisor cluster context

    2. Run below command on supervisor cluster to get the currently assigned IPs for the control plane services.
      kubectl get service -A | awk '/LoadBalancer/ {print $1,$2,$5}' > svcs.txt

       
    3. The above "svcs.txt" file will need to be manually corrected to the original, expected IPs.

      • Steps to Validate from AVI Web UI:

        1. In the AVI web UI, navigate to Operations

        2. Click on Config Audit Trail on the left

        3. With the magnifying glass, search for the name of the affected service

        4. Locate a recent CONFIG_DELETE event and open its details to find the original IP address

          • The next CONFIG_CREATE event has the details of the current, incorrect IP address

      • Steps to Validate Workload Cluster control-plane-services: 

        1. In the Supervisor cluster context, use the below command to get a list of the Control Plane Endpoint IPs for each affected workload cluster:

          kubectl get cluster -o yaml -A | egrep -i "cluster-name|endpoint" -A1

          This endpoint is equivalent to the External IP address of each <cluster-name>-control-plane-service LoadBalancer service for the affected workload cluster(s).

        2. The above output can be compared to the External IP address of the control-plane-service LoadBalancer services for the affected workload cluster(s):
          kubectl get svc -A | grep "control-plane"


    4. Run below command to identify and document the number of replicas for the AKO and net-operator service
      kubectl get deployments -n vmware-system-netop
kubectl get deployments -n vmware-system-ako


    5. Scale down the ako-ako-controller-manager and netop-controller-manager deployment:
      kubectl scale deployment vmware-system-netop-controller-manager -n vmware-system-netop --replicas=0

kubectl scale deployment vmware-system-ako-ako-controller-manager --replicas=0 -n vmware-system-ako


    6. The affected virtual services in the AVI will need to be deleted from the AVI web UI.

      • Navigate to the Avi Load Balancer UI > Templates > Policies > L4 Policy Set in in a web browser

      • Hit the Checkbox for affected Virtual Service's L4 Policy Set (prefix name with cluster name) then click on DELETE in the top left
      • Switch to Virtual Services view in AVI web UI

      • Hit the Checkbox for each affected Virtual Service then click on DELETE in the top left

      • In the ensuing pop-up window, select Delete Virtual Services(VS) and referenced objects

        Note: This deletes Virtual Services(VS) , vsvip , and pool. The l4policyset is a user created object, which has to delete prior the VS and Pool delete.


    7. In the Supervisor cluster context, create the following script:
      vi correct-svcs.sh

#!/usr/bin/bash
filename="$1"
while IFS=' ' read -r f1 f2 f3; do
kubectl get gateway "$f2" -n "$f1" -oyaml | sed 's/addresses: \[\]/addresses:\n    - type: IPAddress\n      value: '"$f3"'/1' | kubectl replace -f -
done < "$filename"

       

    8. Correct the above script's permissions for execution:
      chmod 777 correct-svcs.sh


    9. Run the above script which will iterate through the information in the svcs.txt file created in Step 2 to correct the respective gateway Kubernetes objects:
      ./correct-svcs.sh svcs.txt



    10. Scale the AKO and net-operator pods back up:
      kubectl scale deployment vmware-system-netop-controller-manager -n vmware-system-netop  --replicas=<count from Step 4>

kubectl scale deployment vmware-system-ako-ako-controller-manager --replicas=<count from Step 4> -n vmware-system-ako


    11. Verify that the services are pointed to the correct IPs:
      kubectl get svc -n <virtual service namespace>

       

    12. Confirm that the virtual services in AVI are recreated and stabilize into a healthy state.

  • If the services are not pointed to the correct IPs, you will need to verify what may be still holding the IPs and re-run all of the above steps for the specific services that are still showing as incorrect.

  • If the services are pointed to the correct IPs/you had updated to the correct IPs and the issue continues, it may be the case that IPAM/DNS profiles in AVi are pointed to the noted wrong Network Profile. Make sure they are pointed to the right Network Profile.

Additional Information

Note: If you are using versions above vCenter 8.0u3E and Supervisor cluster version 1.29.7 and stil hitting the issue, you may be hitting this condition:

After an operation that triggers a reconcoliation cycle accross all kubernetes components, such as supervisor upgrade, gateway objects for the guest clusters can be created but may fail to carry over the existing, dynamically leased IP addreses due to the blocking virtual service under AVI.

In that situation you can follow the below resolution. In this case the gateway object is updated with the cluster's controlPlaneEndpoint:

  1. Connect into the Supervisor cluster context

  2. Extract the bound `ControlPlaneEndpoint` IP addresses for all Guest Clusters:
    kubectl get cluster -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.controlPlaneEndpoint.host}{"\n"}{end}'

      
  3. Inspect the corresponding `Gateway` object for each cluster to verify the presence of the `addresses` block
    kubectl get gateway -n <NAMESPACE> <CLUSTER_NAME> -o yaml


  4. Run below command to identify and document the number of replicas for the AKO Pod and net-operator service 
    kubectl get deployments -n vmware-system-netop

kubectl get deployments -n vmware-system-ako


  5. Scale down the AKO and netop-controller-manager deployment:
    kubectl scale deployment vmware-system-netop-controller-manager -n vmware-system-netop --replicas=0

kubectl scale deployment vmware-system-ako-ako-controller-manager --replicas=0 -n vmware-system-ako
  6. Patch the `Gateway` object with the previously identified `ControlPlaneEndpoint` IP:
    kubectl patch gateway -n <NAMESPACE> <CLUSTER_NAME> --type=merge -p '{"spec":{"addresses":[{"type":"IPAddress","value":"<ORIGINAL_IP>"}]}}'

    NOTE: kubectl patch gateway may fail if the IP is already allocated in AVI. In this situation, AVI support is required to assist with manual IPAM adjustments. See "Configuring Avi Load Balancer IPAM".

  7.  Scale up the 'netop-controller' and 'ako' deployments again:
    kubectl scale deployment vmware-system-netop-controller-manager -n vmware-system-netop  --replicas=<count from Step 4>

kubectl scale deployment vmware-system-ako-ako-controller-manager --replicas=<count from Step 4> -n vmware-system-ako
Powered by Wolken Software