Updating private image repository CA certificate to existing clusters.
search cancel

Updating private image repository CA certificate to existing clusters.

book

Article ID: 323913

calendar_today

Updated On:

Products

VMware

Issue/Introduction

For updating private image repository CA certificate to existing legacy clusters.

NOTE: This is applicable to legacy clusters only (not ClusterClass-based clusters).

Symptoms:
There are various components that needs to be updated if you wish to update a custom CA certificate on an already deployed management or workload clusters on Tanzu Kubernetes Grid.

Cause

Updating custom CA certificate to existing clusters.

Resolution

Step 1 - Nodes modification:

     A) On the management cluster context execute the below commands to collect kubeadmconfigtemplates, KubeadmControlPlane and MachineDeployment names. 
kubectl get kubeadmconfigtemplates -A
kubectl get KubeadmControlPlane -A
kubectl get machinedeployment -A
     B) Modify/add the new CA certificate on both kubeadmconfigtemplates and KubeadmControlPlane using the below commands: where KCT-NAME and KCP-NAME are the names acquired from step 1.A. 
kubectl edit KubeadmConfigTemplate KCT-NAME

       spec:
          template:
            spec:
              files:
              - content: |
                  -----BEGIN CERTIFICATE-----
                  MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
                  [...]
                  yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
                  -----END CERTIFICATE-----
                owner: root:root
                path: /etc/ssl/certs/tkg-custom-ca.pem
                permissions: "0644"

kubectl edit KubeadmControlPlane KCP-NAME

       spec:
         kubeadmConfigSpec:
           files:
           - content: |
               -----BEGIN CERTIFICATE-----
               MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
               [...]
               yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
               -----END CERTIFICATE-----
             owner: root:root
             path: /etc/ssl/certs/tkg-custom-ca.pem
             permissions: "0644"
     C) Check preKubeadmCommands block on both KubeadmControlPlane and KubeadmConfigTemplate and add the below commands if not already there. 
 preKubeadmCommands:
       - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
       - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem
          /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'
     D) After modifying the CA certificate, MachineDeployment patching will be needed to start the reconciling the nodes using the below command where MACHINE-DEPLOYMENT is the name acquired from step 1.A. 
kubectl patch machinedeployment MACHINE-DEPLOYMENT --type merge -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"
       Note: Executing this command triggers a rolling update of the cluster nodes and updates their timestamp.

Step 2 - Kapp controller modification for the management cluster.

     A) Edit configmap kapp-controller-config in tkg-system namespace. (update the certificate under data:caCerts:) 
kubectl edit cm kapp-controller-config -n tkg-system

        data:
          caCerts: |
            -----BEGIN CERTIFICATE-----
            MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
            [...]
            yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
            -----END CERTIFICATE-----
     B) Delete Kapp controller pod in tkg-system namespace for reconciliation.


Step 3 - tkr controller modification for the management cluster.

     A) Edit configmap tkr-controller-config in tkr-system namespace. (update the certificate under data:caCerts:) 
kubectl edit cm tkr-controller-config -n tkr-system

       data:
         caCerts: |
           -----BEGIN CERTIFICATE-----
           MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
           [...]
           yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
           -----END CERTIFICATE-----
     B) Delete tkr-controller pod in tkr-system for reconciliation.


Step 4 - kapp controller addon modification for workload cluster. (repeat for all workload clusters)

     A) Decode WORKLOAD-kapp-controller-addon secret value.yaml encrypted section. where WORKLOAD is the name of the workload cluster. 
kubectl get secret WORKLOAD-kapp-controller-addon -o jsonpath="{.data.values\.yaml}" | base64 -d
     B) Modify the certificate. 
       config:
         caCerts: |
           -----BEGIN CERTIFICATE-----
           MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
          [...]
          yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
          -----END CERTIFICATE-----
     C) Base64 Encode the value.yaml modified value.

     D) Edit WORKLOAD-kapp-controller-addon secret by replacing value.yaml value with the modified encoded value.where WORKLOAD is the name of the workload cluster. 
kubectl edit secret WORKLOAD-kapp-controller-addon
      This will reconcile the certificate to the data-value secret and consequently to the kapp configmap on the workload cluster. 

      You can verify by executing the below on the workload cluster. 
kubectl get cm kapp-controller-config -n tkg-system -oyaml


Additional Information

Impact/Risks:
CA certificate authorization between clusters and the private image repository.

Powered by Wolken Software