Rotate Kubernetes Cluster Certificates in Tanzu Kubernetes Grid Integrated fails if the certs are already expired

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

This article describes how to rotate certificates  used by the VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) Kubernetes clusters that are already expired

Running the tkgi rotate-certificates command might fail or can fail during the deployment.

You might also see error similar to below snippet:

Error: 'master/############################(0)' is not running after update.
Review logs for failed jobs: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, vsphere-cloud-controller-manager, csi-controller, csi-syncer, csi-provisioner, csi-attacher, csi-resizer, csi-livenessprobe, csi-snapshotter, blackbox, ncp, fluentd, telegraf, node_exporter, bosh-dns, bosh-dns-resolvconf, bosh-dns-healthcheck, system-metrics-agent.

“missing CA bundle”.

Cause

If the Kubernetes cluster certificates are already expired rotating using tkgi cli would fail with below error 

Error: status 400 reading CredhubClient#regenerateCertificateById(String,CertificateRegenerateRequest,String)

Sometimes it can also fail during the deployment of the nodes.

Resolution

Steps involves running the certificate rotation process using a combination of maestro and credhub which will inturn update the credhub DB, so suggested to reach out to the Broadcom support team. Please include the following info on the support ticket:

1) support bundle

2) maestro tp --name /p-bosh/service-instance_id/kubo_master_ca_2021

3) maestro tp --name /p-bosh/service-instance_id/kubo_ca_2018

4) maestro tp --name /p-bosh/service-instance_id/etcd_ca_2018