How to configure NSX Advanced Load Balancer to provide a load balanced address in front of the kube-vip address for a Tanzu Kubernetes Grid 1.3 cluster

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

When deploying a management or workload cluster on Tanzu Kubernetes Grid (TKG) on vSphere, you may want to access the control plane through a load balancer instead of through the standard kube-vip address.

This could be for reasons such as:

1. The kube-vip address and the cluster are on a non-routable network.
2. You want to expose cluster traffic over standard ports as 443 instead of 6443 (as is the case with kube-vip). 

This article provides instructions for using NSX Advanced Load Balancer (AVI) to configure a load balancer in front of the kube-vip address.

 

Environment

VMware Tanzu Kubernetes Grid 1.x
VMware Tanzu Kubernetes Grid Plus 1.x

Resolution

Prerequisites:

• An AVI installation to be used for creating the load balancer in front of kube-vip.

Notes:

• This procedure should work with other load balancing solutions but in that case you would have to refer to the product documentation to create the required load balancer.
• The proposed solution creates an AVI load balancer in front of the kube-vip / cluster control plane. In case of existing clusters, it utilizes the Cluster API Kubeadm control plane controller to trigger a rollout of the control plane nodes with a newly generated server certificate for the API server. For new clusters, a vsphere overlay will ensure that the load balancer IP address is included in the API server certificate. 

Update Existing Clusters:

If you have already installed the TKG management cluster and one or more workload clusters you can use the following procedure to add an AVI load balancer in front of kube-vip / control plane.

1. With your kubectl context set to the management cluster, issue a command similar to the following to get the current IP address for the cluster control plane endpoint:

kubectl get cluster <cluster name> -o=jsonpath='{.spec.controlPlaneEndpoint}{"\n"}' (also pass -n tkg-system to if updating a management cluster)

Note: You will see output similar to the following:

{"host":"192.###.###.###","port":6443}

2. In the AVI UI, navigate to Applications, Pools and click the Create Pool button.
3. Complete Step 1 of the New Pool wizard per the following:

Name: <cluster name>-controlplane-pool

Default Server Port: 6443
 

Note: All other fields can be left at default values

Click the Next button.

4. Complete Step 2 of the New Pool wizard per the following:

Server IP Address: Enter the IP address noted in Step 1 (192.###.###.### in this example).
Click the Add Server button.

Note: All other fields can be left at default values.

Click the Next button.

5. Click the Next button on Step 3 of the New Pool wizard as there are no changes needed on that page.
6. Click the Save button on Step 4 of the New Pool wizard.
7. Navigate to the Virtual Services tab and click on the Create Virtual Service dropdown and then select Advanced Setup.
8. Complete Step 1 of the New Virtual Service wizard per the following:

Name: <cluster name>-controlplane-lb
Application Profile: System-L4-Application
Network for VIP Address Allocation: Select the appropriate frontend network
IPv4 Subnet: Select the appropriate IPv4 subnet.
Services: 443
Pool: Select the pool created in Steps two through six.

Note: All other fields can be left at default values.

Click the Save button.

9. Click the Next button on Step 2 and Step 3 of the New Virtual Service wizard as there are no changes needed on that page..
10. Click the Save button on Step 4 of the New Virtual Service wizard.
11. Navigate back to the Virtual Services tab and make a note of the IP address in the Address field for the <cluster name>-controlplane-lb Virtual Service (192.###.###.# in this example).
12. With your kubectl context set to the appropriate cluster, edit the kubeadmcontrolplane object by issuing a command similar to the following:

kubectl edit kubeadmcontrolplane <cluster name>-control-plane (also pass -n tkg-system to if updating a management cluster)

Find the spec: kubeadmConfigSpec: clusterConfiguration: apiSever: section and add two lines similar to the following:

        certSANs:
        - <IP address noted in Step 11>

Note: The kubeadmConfigSpec stanza should look similar to the following:

  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        certSANs:
        - 192.###.###.#

Note: You can replace the IP address with a fully qualified domain name that maps to the IP address.

13. Type :wq to save the change.

Note: You should see a template being cloned in the vSphere Client as new control plane nodes are created. You will also see the original control plane nodes being deleted. When only the new nodes remain, proceed to the next step.

14. To connect to this cluster over the AVI load balancer address as the admin user, you have two options.
    1. If you have a combined kubeconfig file you will need to run a command similar to kubectl config delete-context <cluster context name> and then run tanzu cluster kubeconfig get <cluster name> --admin (also pass -n tkg-system to get a management cluster kubeconfig).
    2. If you use a separate file for each cluster, run a command similar to tanzu cluster kubeconfig get <cluster name>  --admin --export-file <kubeconfig file> (also pass -n tkg-system to get a management cluster kubeconfig). In both scenarios, edit the kubeconfig file and replace the kube-vip address (192.###.###.###  in this example) with the AVI load balancer address (192.###.###.# in this example) and remove :6443 from the server: line.

Note: You can replace the IP address with a fully qualified domain name that maps to the IP address.

14. To connect to this cluster over the AVI load balancer address as an LDAP or OIDC user, you need to run a command similar to tanzu cluster kubeconfig get <cluster name> --export-file <kubeconfig file> (also pass -n tkg-system to get a management cluster kubeconfig file). Edit the kubeconfig file and replace the kube-vip address (192.###.###.### in this example) with the AVI load balancer address (192.###.###.# in this example) and remove :6443 from the server: line.

Note: You can replace the IP address with a fully qualified domain name that maps to the IP address.

Create New Clusters:

1. Follow Steps two through eleven from the Update Existing Clusters section to create the needed Pool and Virtual Server. Supply the desired kube-vip address on Step 4.
2. Add content similar to the following to the ~/.tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-overlay.yaml file: 

#@overlay/match by=overlay.subset({"kind":"KubeadmControlPlane"})

---

apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: KubeadmControlPlane
metadata:
  name: #@ "{}-control-plane".format(data.values.CLUSTER_NAME)
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
     apiServer:
      certSANs:
      - 192.###.###.#

Note: Replace the IP address with the desired the AVI load balancer address. This file will need to be updated each time a new cluster is created so that the same IP address is unique for each cluster.
Note: You can replace the IP address with a fully qualified domain name that maps to the IP address.

3. Create your management or workload cluster as usual via the tanzu CLI.
4. Follow Steps 14 and 15 from the Update Existing Clusters section to create and modify the kubeconfig file for the new cluster.