Manually renew cluster certificates for TKGm clusters deployed via TCA 3.2

Products

VMware Telco Cloud Automation

Issue/Introduction

Cluster VIP is inaccessible due to TKGm cluster certificate expiration

Environment

3.2

Cause

TCA supports automatic certificate renewal for both Kubernetes management cluster and v2 workload clusters. However, in some cases, it may be required to manually renew the cluster certificates and/or kubeconfig stored in the TCA database.

Resolution

Checking if Cluster Certificates are valid:

Query the Kubernetes Cluster Control Plane node for the status and expiration date of the Kubernetes Cluster certificates:

SSH to the Control Plane node of the CaaS Cluster and switch to the sudo user:

ssh capv@K8S-CONTROL-PLANE-IP

sudo -i
Check if certificates are expired:

kubeadm certs check-expiration

Another verification point is to login to each TCA-CP Appliance Management UI and looking at the status of the Kubernetes Clusters registered within TCA-CP:

A green dot would mean that the communication is fine and there is no certificate related issue.
A red dot would mean that the communication is broken and there could be a possible certificate related issue.

Renewing the Workload Cluster Certificate (Don't use root)

Note: The following steps are applicable only for TKG clusters that were deployed via older versions of TCA and have NOT been upgraded in TCA 3.2.

SSH into the TCA-CP as admin where corresponding management cluster is deployed.
Download the attached cluster-cert-renew-TCA3.2.tar.gz script file.
Untar the cluster-cert-renew-TCA3.2.tar.gz tar ball:

tar -zxvf cluster-cert-renew-TCA3.2.tar.gz
Renew the workload cluster certificate:

cd /home/admin/cluster-cert-renewbash cert-renew -wc workload-cluster-name -mc mgmt-cluster-name -t workload

Note: Replace workload-cluster-name and mgmt-cluster-name with the actual values in the command provided.
Verify the new workload cluster certificate has been stored on management cluster:

kubectl config use-context mgmt-cluster-name-admin@mgmt-cluster-name

kubectl get secret workloadcluster-name-kubeconfig -n workloadcluster-name -ojsonpath='{.data.value}' | base64 -d
Note: Replace mgmt-cluster-name-admin, mgmt-cluster-name and workloadcluster-name with the actual values in the command provided.

Renew the Management Cluster Certificates (Don't use root)

SSH into the TCA-CP as admin where corresponding management cluster is deployed.
Download the attached cluster-cert-renew-TCA3.2.tar.gz script file.
Untar the cluster-cert-renew-TCA3.2.tar.gz tar ball and change to the cluster-cert-renew directory:

tar -zxvf cluster-cert-renew-TCA3.2.tar.gz

cd /home/admin/cluster-cert-renew
Identify the management cluster control plane node where kube-vip pod is scheduled.

ssh to each control plane node
sudo -i

crictl ps | grep -i vip
From TCA CP renew the management cluster certificate using the control plane node ip identified above.

bash cert-renew -mc mgmt-cluster-name -t management -ip control-plane-node-ip

Note: This can take several minutes to complete.

Synchronize the kubeconfig for the TCA-Manager (TCA-M) and TCA-Control Plane (TCA-CP)

Note: All (upgraded and non-upgraded) Clusters require the kubeconfig to be synchronized.

Note: Steps 1 through 4 should only be applied to TCA-M and not TCA-CP.

To generate an authentication token on TCA 3.2, you must POST the following API call from a seperate Linux VM that has access to the TCA-M:

curl -D - --location --insecure --request POST 'https://tca-m-url/hybridity/api/sessions' --header 'Accept: application/json' --header 'Content-Type: text/plain' --data-raw '{"username": "username","password": "plain_text_password"}'Note:

Replace tca-m-url, username and plain_text_password with the actual values in the command provided.
Take note of the x-hm-authorization UUID from the output of the previous step.
Update the TCA-M and TCA-CP database by synchronizing the kubeconfig:

curl --location --insecure --request POST 'https://tca-m-fqdn/telco/api/caas/v2/clusters/cluster_name/syncKubeconfig' --header 'Accept: application/json' --header 'Content-Type: application/json' --header 'x-hm-authorization: auth-token'

Note: Replace tca-m-fqdn, cluster_name and auth-token with the actual values in the command provided.

Note: The operation can take several minutes.
To ensure that the operation is succeeded, run the following API call:

curl --location --insecure --request GET 'https://tca-m-fqdn/hybridity/api/jobs/job_id_from_above_response' --header 'Accept: application/json' --header 'x-hm-authorization: auth-token'

Note: Replace tca-m-fqdn, auth-token and job_id_from_above_response with the actual values in the command provided.

Note: Take note of the isDone and didFail flags in the json. The isDone flag should return true and the didFail flag should return false.
SSH login to TCA-CP to restart the services:

ssh admin@tca-cp

Note: tca-cp where the cluster is configured
Restart tca-app service on tca-cp :

kubectl rollout restart deployment tca-app -n tca-cp-cn

Note: In case of multiple TCA-CPs (i.e one for mgmt cluster & one for workload cluster) the app service should be started from both.

Additional Information

IPv6 endpoint cluster is not supported
Certificate renewal is automated and expected to run as designed so the recommendation is to wait for the process to trigger. Use of this KB is recommended in scenarios where auto renewal fails to rotate the certificates. During the auto renewal of certificate procedure ideally there is no impact on the application itself. Only new control-plane nodes are rolled out during renewal process of the certificates with no impact on the worker nodes. The Kubelet certificates on worker nodes will be renewed by itself only at their expiry with a auto restart of kubelet service. In both cases, e.g. after successful auto renewal Or manual renewal the new kubeconfig needs to be updated in other integrated systems dependent on the kubeconfig for connectivity e.g. jump server etc.

Attachments

cluster-cert-renew-TCA3.2.tar.gz get_app