How to Force a Cluster Configuration Update/Manual Rolling Redeployment of Nodes in a vSphere Kubernetes Cluster on ClusterClass v3.3.X

Products

• VMware vSphere Kubernetes Service
• vSphere with Tanzu
• Tanzu Kubernetes Runtime

Issue/Introduction

This KB is written to advise on how to force propagate node rolling redeployment in vSphere Kubernetes clusters on ClusterClass v3.3.X for a vSphere Kubernetes Supervisor environment.

A rolling redeployment can be triggered by cluster edits such as changing the vmclass, editing volume mounts or upgrading the TKR.

This KB is intended for initiating a rolling redeployment without making any changes to the cluster.

The steps below are intended for scenarios where an automatic rolling update is not triggered after a configuration change which requires nodes to be recreated with the new desired configuration. For example:

• Adding a new instance of or editing an instance of Harbor as a Supervisor Service
  • Installing Harbor as a Supervisor Service
  • Updating an existing Harbor Supervisor Service's certificates
    
    
• Editing one of the following cluster variables:
  • osConfiguration.systemProxy
  • osConfiguration.trust
  • osConfiguration.domainJoin
  • osConfiguration.sshd.banner
  • resourceConfiguration.systemReserved

Environment

• vSphere with Tanzu 8.0
• VKS Service/TKG Service (Supervisor Service) v3.3.0, v3.3.1 or v3.3.2
• vSphere Kubernetes Cluster using clusterclass builtin-generic-v3.3.X

Cause

• In VKS/TKG Service v3.3.0, clusters using clusterclass builtin-generic-v3.3.0 store their configuration within a Secret.

  • This Secret is referenced by a KubeadmConfig object, which is individually associated with each node.

  • When a configuration change occurs, the VKS/TKG service directly modifies the contents of this existing Secret without any change to it name.

  • Because the Secret's name remains unchanged during the update, the new configuration fails to automatically propagate from the Secret to the nodes.

  • As such, these configurations require a manually triggered rolling redeployment to propagate to the updated cluster's nodes.

Resolution

In order to ensure that the new configuration is propagated to the new nodes, a rolling redeployment needs to be initiated on the updated cluster.

This KB advises on steps to initiate a rolling redeployment without making any changes to the cluster.

A rolling redeployment can be performed by adding a new variables labels section in the cluster yaml.

1. Connect into the Supervisor cluster context
   
   
2. Edit the updated cluster's cluster object:

   • kubectl edit cluster <updated cluster name> -n <updated cluster namespace>

3. Edit the node.labels variable to perform a cluster-wide rolling redeployment of nodes:
   • Add the following section to the cluster object's variables section, where manual can be replaced by a valid Kubernetes label:

     • topology:
            class: builtin-generic-v3.3.#
            version: v#.##.#---vmware.#-fips-tkg.#
            controlPlane:
              replicas: #
            workers:
              machineDeployments:
                - class: node-pool
                  name: node-pool-1
                  replicas: #
            variables:
              - name: vmClass
                value: guaranteed-medium
              - name: storageClass
                value: my-storage-policy
              - name: defaultStorageClass
                value: my-storage-policy
              - name: node
                value:
                  labels:
                    update: manual

4. Ensure that a rolling redeployment has initiated for all nodes in the cluster:
   • Note: Updating or removing the above section again will queue another rolling redeployment.

   • kubectl get kcp,md,ma -n <updated cluster namespace>

5. Wait for the rolling redeployment to complete.
   
   
6. Confirm that the desired configuration change was successful on the new nodes.

Additional Information

The added section will remain until manually removed from the cluster YAML.

Note: Another rolling redeployment will not be triggered by this section until it is manually changed to a different update: value or unless this added section is removed from the cluster's YAML.

Automatic rolling redeployment after configuration changes will be available starting in VKS/TKG Service 3.3.3.