Updating private image repository CA certificate to existing clusters.

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x
• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

For updating private image repository CA certificate to existing legacy clusters.

NOTE: This is applicable to legacy clusters only (not ClusterClass-based clusters).

Symptoms:
There are various components that needs to be updated if you wish to update a custom CA certificate on an already deployed management or workload clusters on Tanzu Kubernetes Grid.

Environment

Tanzu Kubernetes Grid Multi-Cloud

Note: This is not a valid kb for vSphere with Tanzu/TKGS.

Cause

Updating custom CA certificate to existing clusters.

Resolution

Step 1 - Nodes modification:

     A) On the management cluster context execute the below commands to collect kubeadmconfigtemplates, KubeadmControlPlane and MachineDeployment names.

kubectl get kubeadmconfigtemplates -A
kubectl get KubeadmControlPlane -A
kubectl get machinedeployment -A

     B) Modify/add the new CA certificate on both kubeadmconfigtemplates and KubeadmControlPlane using the below commands: where KCT-NAME and KCP-NAME are the names acquired from step 1.A.

kubectl edit KubeadmConfigTemplate KCT-NAME

       spec:
          template:
            spec:
              files:
              - content: |
                  -----BEGIN CERTIFICATE-----
                  MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
                  [...]
                  yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
                  -----END CERTIFICATE-----
                owner: root:root
                path: /etc/ssl/certs/tkg-custom-ca.pem
                permissions: "0644"

kubectl edit KubeadmControlPlane KCP-NAME

       spec:
         kubeadmConfigSpec:
           files:
           - content: |
               -----BEGIN CERTIFICATE-----
               MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
               [...]
               yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
               -----END CERTIFICATE-----
             owner: root:root
             path: /etc/ssl/certs/tkg-custom-ca.pem
             permissions: "0644"

     C) Check preKubeadmCommands block on both KubeadmControlPlane and KubeadmConfigTemplate and add the below commands if not already there.

 preKubeadmCommands:
       - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
       - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem
          /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'

     D) After modifying the CA certificate, MachineDeployment patching will be needed to start the reconciling the nodes using the below command where MACHINE-DEPLOYMENT is the name acquired from step 1.A.

#For TKG 2.2 and older (ClusterAPI v1.3 and older):
kubectl patch machinedeployment MACHINE-DEPLOYMENT --type merge -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

#For TKG 2.3 and newer (ClusterAPI v1.4 and newer):
kubectl patch machinedeployment MACHINE-DEPLOYMENT --type merge -p "{\"spec\":{\"rolloutAfter\":\"$(date +'%Y-%m-%dT%TZ')\"}}"

       Note: Executing this command triggers a rolling update of the cluster nodes and updates their timestamp.

Step 2 - Kapp controller modification for the management cluster.

     A) Edit configmap kapp-controller-config in tkg-system namespace. (update the certificate under data:caCerts:)

kubectl edit cm kapp-controller-config -n tkg-system

        data:
          caCerts: |
            -----BEGIN CERTIFICATE-----
            MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
            [...]
            yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
            -----END CERTIFICATE-----

     B) Delete Kapp controller pod in tkg-system namespace for reconciliation.


Step 3 - tkr controller modification for the management cluster.

     A) Edit configmap tkr-controller-config in tkr-system namespace. (update the certificate under data:caCerts:)

kubectl edit cm tkr-controller-config -n tkr-system

       data:
         caCerts: |
           -----BEGIN CERTIFICATE-----
           MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
           [...]
           yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
           -----END CERTIFICATE-----

     B) Delete tkr-controller pod in tkr-system for reconciliation.


Step 4 - kapp controller addon modification for workload cluster. (repeat for all workload clusters)

     A) Decode WORKLOAD-kapp-controller-addon secret value.yaml encrypted section. where WORKLOAD is the name of the workload cluster.

kubectl get secret WORKLOAD-kapp-controller-addon -o jsonpath="{.data.values\.yaml}" | base64 -d

     B) Modify the certificate.

       config:
         caCerts: |
           -----BEGIN CERTIFICATE-----
           MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
          [...]
          yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
          -----END CERTIFICATE-----

     C) Base64 Encode the value.yaml modified value.

     D) Edit WORKLOAD-kapp-controller-addon secret by replacing value.yaml value with the modified encoded value.where WORKLOAD is the name of the workload cluster.

kubectl edit secret WORKLOAD-kapp-controller-addon

      This will reconcile the certificate to the data-value secret and consequently to the kapp configmap on the workload cluster. 

      You can verify by executing the below on the workload cluster.

kubectl get cm kapp-controller-config -n tkg-system -oyaml

 

       If the configmap is not getting updated in the workload cluster, then check the status of the reconciliation of the workload cluster's kapp-controller app in the Management Cluster.  You can run these commands using the Management Cluster context to verify.  If it is failing, then you need to address the issue accordingly first and then verify the configmap again afterwards.

kubectl get apps -n NAMESPACE WORKLOAD-kapp-controller
kubectl describe apps -n NAMESPACE WORKLOAD-kapp-controller

Additional Information

Impact/Risks:
CA certificate authorization between clusters and the private image repository.