Manually renew cluster certificates

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Cluster vip inaccessible

Environment

TCA: 2.3

Cause

Starting in 2.3, TCA supports automatic certificate renewal for both the Kubernetes management cluster and v2 workload clusters. In some cases, it may be required to manually renew the cluster certificates and/or kubeconfig stored in the TCA database.

Resolution

Checking if Cluster Certificates are valid:
Run the following commands to query the Kubernetes Cluster Control Plane node for the status and expiration date of Kubernetes Cluster certificates:

1. SSH to the Control Plane node of the CaaS Cluster and switch to the sudo user:
   ssh capv@K8S-CONTROL-PLANE-IP
   
   
2. Switch to root
   sudo -i
   
   
3. Check if certificates are expired by
   kubeadm certs check-expiration

 

Updating the Cluster certificate within Caas

1. TCA 2.3 introduces the automatic renewal of the Cluster Certificates, given the CaaS Clusters have been upgraded to the supported Tanzu Kubernetes Grid (TKG) clusters: 1.24.10, 1.23.16 and 1.22.17.
   
   
2. Clusters upgraded to TKG 1.24.10, 1.23.16 or 1.22.17:
   
   
3. TCA 2.3 will automatically regenerate the Kubernetes cluster certificates. However, these clusters will still need their kubeconfig updated. 
   
   Note: The following steps apply only to TKG clusters deployed with older versions of TCA and NOT upgraded to TCA 2.3.

Renewing the Workload Cluster Certificate

1. SSH into the TCA-CP where the corresponding management cluster is deployed as the admin user and switch to the sudo user:
   ssh admin@mgmt_cluster_tca_cp_fqdn
su –
   
   Note: Replace mgmt_cluster_tca_cp_fqdn with the actual values in the command provided.
   
   
2. Download the attached certificate renewal tool file: tca2.3-cluster-cert-renew.tar.gz.
   
   Note: For Airgap environments, manually SCP the file over to the TCA-CP:
   
   
3. Untar the tca2.3-cluster-cert-renew.tar.gz:
   tar -zxvf tca2.3-cluster-cert-renew.tar.gz
   
   
4. Renew the workload cluster certificate:
   cd /home/admin/cluster-cert-renew
   bash cert-renew -wc workload-cluster-name -mc mgmt-cluster-name -t workload
   
   Note: Replace workload-cluster-name and mgmt-cluster-name with the actual values in the command provided.
   
   
5. Verify the new workload cluster certificate has been stored on the management cluster:
   kubectl config use-context mgmt-cluster-name-admin@mgmt-cluster-name
   kubectl get secret workloadcluster-name-kubeconfig -n workloadcluster-name -ojsonpath='{.data.value}' | base64 -d
   
   Note: Replace mgmt-cluster-name-admin and mgmt-cluster-name and workloadcluster-name with the actual values in the command provided.

Renew the Management Cluster Certificates

1. SSH into the TCA-CP and switch to the sudo user:
   ssh admin@tca_cp
su –

Note: Replace tca_cp  with the IP of the TCA-CP where the management cluster is configured in the command provided. 

2. Download the cluster-cert-renew scripts tar to TCA-CP:
   curl -kfsSL https://packages.broadcom.com/artifactory/tca-distro/kb/20230413/cluster-cert-renew.tar.gz

3. Untar the cluster-cert-renew.tar.gz and change to the cluster-cert-renew directory:
   tar -zxvf cluster-cert-renew.tar.gz
cd /home/admin/cluster-cert-renew 

4. Obtain the Control Plane node IP
   kubectl get nodes -owide | grep control-plane | awk '{print ""$6""}' | head -n 1

5. Renew the management cluster certificate.
   bash cert-renew -mc mgmt-cluster-name -t management -ip control-plane-node-ip

Synchronize the kubeconfig for the TCA-Manager (TCA-M) and TCA-CP.

1. To generate an authentication token on TCA 2.3, you can POST the following API call from any machine that has access to the TCA-M web layer. This can be executed from the TCA-M Manager, TCA-CP, or any other Linux VM with access to the TCA-Manager.
   
   Note: Steps 1 through 4 should be applied only to TCA-M, not to TCA-CP.
   
   
2. curl -D - --location --insecure --request POST 'https://tca-m-url/hybridity/api/sessions' --header 'Accept: application/json' --header 'Content-Type: text/plain' --data-raw '{"username": "username","password": "plain_text_password"}'
   
   Note: Replace tca-m-url and username and plain_text_password
   
   
3. Take note of the x-hm-authorization UUID from the output of the previous step.
   
   
4. Update the TCA-M and TCA-CP database by synchronizing the kubeconfig:
   curl --location --insecure --request POST 'https://tca-m-fqdn/telco/api/caas/v2/clusters/cluster_name/syncKubeconfig' --header 'Accept: application/json' --header 'Content-Type: application/json' --header 'x-hm-authorization: auth-token'
   
   Note: Replace tca-m-fqdn and cluster_name and auth-token with the actual values in the command provided.
   
   
5. To ensure that the operation succeeds, run the following API call:
   curl --location --insecure --request GET 'https://tca-m-fqdn/hybridity/api/jobs/job_id_from_above_response' --header 'Accept: application/json' --header 'x-hm-authorization: auth-token'
   
   Note: Replace tca-m-fqdn, auth-token, and job_id_from_above_response with the actual values in the command provided.
   Note: Take note of the isDone and didFail flags in the json. The isDone flag should return true and the didFail flag should return false.
   
   
6. SSH login to TCA-CP to restart the services:
   ssh admin@tca-cp
   su -
   
   Note: tca-cp where the cluster is configure
    
7. Restart the following TCA-CP services:

systemctl restart app-engine
systemctl restart web-engine

Note:
All (upgraded and non-upgraded) Clusters require the kubeconfig to be synchronized.
In case of multiple TCA-CPs (i.e one for Mgmt cluster & one for Workload cluster) the app & web services should be started from both.

 

Additional Information

• See Create a v2 Workload Cluster Template 
  
  
• Clusters deployed by TCA 2.3 and currently managed by the TCA 2.3 version.

• The automatic renewal of TKG cluster certificates can fail for various reasons, one of which is described below.

• For the "Control Plane Node Certificate Auto-Renewal" feature to function correctly, new nodes must be rolled out. If any node becomes stuck in a provisioning state, the renewal process will be unsuccessful.

• During certificate renewal, control plane nodes are replaced sequentially, with each new node joining the cluster only after the preceding one is fully operational, ensuring the etcd quorum remains intact.

• Certificate renewal may also fail due to issues with etcd or if a control plane node is unable to complete provisioning.

• Latest Versions of TCA 3.x now offers enhanced certificate observability , management, and TCA 3.4 includes an automated workflow for manual rotation, allowing for one-click certificate refresh and verification.

• References:
  
  TCA 3.4 Certificate Observability & Management Feature

  TCA 3.1 Release Notes