Rotate TKG classy cluster’s ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE/ADDITIONAL_IMAGE_REGISTRY_2_CA_CERTIFICATE/ADDITIONAL_IMAGE_REGISTRY_3_CA_CERTIFICATE certificates

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Limitations
This certificate rotation procedure is only supported when the cluster is still in a healthy state and the certificate hasn’t expired. Since if the cluster is not in a good state, and the certificate expired, the node might have issues coming up since containerd can’t pull the image.

We don’t have docs to rotate expired certificates for now.

Symptoms:

Currently TKG support those parameters to configure external certificates in classy clusters:

• TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE
• TKG_PROXY_CA_CERT
• CUSTOM_TDNF_REPOSITORY_CERTIFICATE
• ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE(TKG 2.2)
• ADDITIONAL_IMAGE_REGISTRY_2_CA_CERTIFICATE(TKG 2.2)
• ADDITIONAL_IMAGE_REGISTRY_3_CA_CERTIFICATE(TKG 2.2)

This document contains steps on how to rotate ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE, ADDITIONAL_IMAGE_REGISTRY_2_CA_CERTIFICATE, ADDITIONAL_IMAGE_REGISTRY_3_CA_CERTIFICATE.

Since these are external certificates, the most straightforward solution is the traditional 3-steps certificate rotation method. In general:

1. The customer updates the cluster variable with old and new certificates encoded in a single string. CAPI starts to roll out the cluster which gets the update.
2. The customer updates the registry or proxy he/she wants to update the certificate - proxy, repository, tdnf repository - with new certificates.
3. The customer updates the cluster variable with only a new certificate. CAPI starts to roll out the cluster which gets the update.

Environment

VMware Tanzu Kubernetes Grid 2.2.0
VMware Tanzu Kubernetes Grid 2.3.0
VMware Tanzu Kubernetes Grid 2.1.0
VMware Tanzu Kubernetes Grid 2.1.1
VMware Tanzu Kubernetes Grid 2.4.0

Resolution

Phase 1

1. Update management cluster
   1. Updates `cluster.spec.topology.variables.trust.additionalImageRegistries[0].caCert` with both new and old certificates base64 encoded together.

For ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE, update cluster.spec.topology.variables.trust.additionalImageRegistries[0].caCert,
For ADDITIONAL_IMAGE_REGISTRY_2_CA_CERTIFICATE, update cluster.spec.topology.variables.trust.additionalImageRegistries[1].caCert,
For ADDITIONAL_IMAGE_REGISTRY_3_CA_CERTIFICATE, update cluster.spec.topology.variables.trust.additionalImageRegistries[2].caCert

1. For example, put below into a single file

-----BEGIN CERTIFICATE-----
Old
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
New
-----END CERTIFICATE-----

 

2. Encode the file with base64. This base64 value will be also used later

$ cat <certifcate-file-name> | base64 -w0
## use below string to represent the encoded value in following examples
AAAAB3NzaC1yc2EAAAADAQABAAAA1

 

3. Update the management cluster object

kubectl edit <cluster-name> -n <namespace>

 

4. This will trigger cluster rollout, it might take a while. Monitor cluster rollout

kubectl get nodes

 

2. Updates tkg-pkg with base64 encoded old and new certificates, generated in step 1.ii

1. First get the content and save to a file, for example tkg-pkg-values

kubectl get secret -n tkg-system tkg-pkg-tkg-system-values -ojsonpath='{.data.tkgpackagevalues\.yaml}' | base64 -d > tkg-pkg-values

 

2. Update the content configvalues.ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE, configvalues.ADDITIONAL_IMAGE_REGISTRY_2_CA_CERTIFICATE and configvalues.ADDITIONAL_IMAGE_REGISTRY_3_CA_CERTIFICATE with encoded base64 certificates

configvalues:
  ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE: AAAAB3NzaC1yc2EAAAADAQABAAAA1
  ## if set
  ADDITIONAL_IMAGE_REGISTRY_2_CA_CERTIFICATE: AAAAB3NzaC1yc2EAAAADAQABAAAA2
  ## if set
  ADDITIONAL_IMAGE_REGISTRY_3_CA_CERTIFICATE: AAAAB3NzaC1yc2EAAAADAQABAAAA3

 

3. Encode the file again with base64 command

cat tkg-pkg-values | base64 -w0

 

4. Update the secret tkg-pkg-tkg-system-values with new value generated above. Kapp controller will pick up correct certificates from the cluster’s annotations, so no need to modify its configuration

kubectl edit secret -n tkg-system tkg-pkg-tkg-system-values
## edit .data.tkgpackagevalues.yaml

 

2. Update workload cluster
   1. Repeat step 1.a, except targeting the workload cluster’s object

 

Phase2

Update the certificate on the registry server that the customer manage

Phase3

Redo the phase1, except, this time only encodes the new certificate, no need to use the old certificate.

 

Verification

1. Cluster could rollout in Phase 3
2. Make sure no error in the kapp-controller in MC and WC
3. Make sure no Error in containerd on VM node, pod that using image from these registries could be deployed without image pulling error