Installing or Re-installing Harbor as a Supervisor Service triggers a rollout of all the Guest Cluster nodes

Products

• Tanzu Kubernetes Runtime
• VMware vSphere Kubernetes Service

Issue/Introduction

• Installing or re-installing Harbor as a supervisor service triggers a rollout of all the Guest Cluster nodes.
  
  
•  Logs of the capi-kubeadm-control-plane-controller-manager pod confirm that the nodes were rolled out because of the outdated KubeadmConfig of the cluster nodes.
  
  I1111 1 controller.go] "Rolling out Control Plane machines: Machine <name of the machine> needs rollout: Machine KubeadmConfig InitConfiguration or JoinConfiguration are outdated: diff: &v1beta1.KubeadmConfigSpec{\n    ClusterConfiguration: nil,\n    InitConfiguration:    &{NodeRegistration: {KubeletExtraArgs: {\"cloud-provider\": \"external\", \"event-qps\": \"0\", \"node-ip\": \"{{ ds.meta_data.local_ipv4 }}\", \"node-labels\": \"run.tanzu.vmware.com/tkr=v1.32.0---vmware.6-fips-vkr.2,run.tanzu\"..., ...}, ImagePullPolicy: \"IfNotPresent\"}},\n    JoinConfiguration:    nil,\n    Files: []v1beta1.File{\n      ... // 6 identical elements\n      {Path: \"/etc/ssl/certs/extensions-tls.crt\", Owner: \"root:root\", Permissions: \"0644\", ContentFrom: &{Secret: {Name: \"cluster1-extensions-ca\", Key: \"tls.crt\"}}},\n      {Path: \"/run/machine-agent/ca-cert\", Owner: \"root:root\", Permissions: \"0644\", ContentFrom: &{Secret: {Name: \"cluster1-ma-token\", Key: \"machine-agent-ca-cert\"}}},\n      {\n        ... // 4 identical fields\n


• The describe output of the cluster object also confirms that the KubeadmConfig is outdated.
  
        message: |-
        * MachineDeployment <nodepool-name>:
          * Rolling out <number of replicas> not up-to-date replicas
            * KubeadmConfig is not up-to-date
      observedGeneration: <number>
      reason: RollingOut
      status: "True"
      type: RollingOut

Environment

vSphere Kubernetes Service 3.3.3 and later

Cause

Installing or re-installing Harbor Supervisor Service updates the required certificate in the bootstrap information of the existing VKS clusters and therefore a cluster-wide rollout is needed.

Resolution

This is an expected behavior. In the event Harbor is deployed or re-deployed, a cluster-wide rollout must be triggered automatically.

Note: In the event the cluster rollout doesn't happen automatically, a manual rollout has to be triggered to propagate the updated configuration to the cluster nodes. Refer the following kb article on how to achieve the same- How to Force a Cluster Configuration Update/Manual Rolling Redeployment of Nodes in a vSphere Kubernetes Cluster on ClusterClass v3.3.X