TKGm Certificate Rotation - Tanzu CLI and kubectl

Products

• VMware Tanzu Kubernetes Grid
• Tanzu Kubernetes Runtime

Issue/Introduction

When the Control Plane node (CP node) certificates have been rotated but the kubectl via jumpbox does not function, you must update the client certificate in your kubeconfig file.

This KB describes how to update the kubeconfig using the Tanzu CLI after rotating CP node certificates.

kubectl get nodes
#> Unable to connect to the server: x509: certificate has expired or is not yet valid

Check the client-certificate of Tanzu CLI in the jumpbox.

# Set Management Cluster name
MC=mgmt-v254
cat ~/.kube-tkg/config | yq ".users[]|select(.name==\"${MC}-admin\").user.client-certificate-data" | base64 -d | openssl x509 -noout -serial -dates

Check the client-certificate of kubectl target cluster context in the jumpbox.

# Set the target cluster name
CLUSTER=mgmt-v254
cat ~/.kube/config | yq ".users[]|select(.name==\"${CLUSTER}-admin\").user.client-certificate-data" | base64 -d | openssl x509 -noout -serial -dates

Environment

• TKGm v2.1.x
• TKGm v2.2.x
• TKGm v2.3.x
• TKGm v2.4.x
• TKGm v2.5.x

Resolution

Update Tanzu CLI client-certificate

1. Set the Management Cluster name

# If the client-cert in the tanzu CLI is not expired, this command works
tanzu cluster list --include-management-cluster -A
MC=mgmt-v254

# Check the tanzu CLI client cert
cat ~/.kube-tkg/config | yq ".users[]|select(.name==\"${MC}-admin\").user.client-certificate-data" | base64 -d | openssl x509 -noout -serial -dates

2. Backup

cp ~/.kube-tkg/config backup-kube-tkg-config-$(date "+%Y%m%d%H%M")

3. Retrieve the latest kubeconfig from CP node 

KCP=192.168.x.x
ssh capv@${KCP} sudo cat /etc/kubernetes/admin.conf | yq ".users[0].user"
#> client-certificate-data: XXXXXXXXXXX...XXX
#> client-key-data: YYYYYYYYYYY...YYY

4. Update the ~/.kube-tkg/config manually under the target Management Cluster section

vi ~/.kube-tkg/config
- Update client-certificate-data: XXXXXXXXX...XXX
- Update client-key-data: YYYYYYYYYYY...YYY

# Check the format for make sure
cat ~/.kube-tkg/config | yq ".users[]|select(.name==\"${MC}-admin\").user.client-certificate-data" | base64 -d | openssl x509 -noout -serial -dates

5. Check the behavior of Tanzu CLI

tanzu context list
tanzu context use ${MC}
tanzu mc get ${MC}

 

Update kubectl client-certificate for Management Cluster

1. (jumpbox) Set the Management Cluster name

tanzu cluster list --include-management-cluster -A
MC=mgmt-v254

# Check the kubectl client cert of Management Cluster context
cat ~/.kube/config | yq ".users[]|select(.name==\"${MC}-admin\").user.client-certificate-data" | base64 -d | openssl x509 -noout -serial -dates

2. (jumpbox --> CP node) SSH to the one of the CP node

KCP=192.168.x.x
ssh capv@${KCP}
sudo -i
export KUBECONFIG=/etc/kubernetes/admin.conf
# Set the Management Cluster name again
MC=mgmt-v254

3. (CP node) Patch the secret

encoded_data=$(cat /etc/kubernetes/admin.conf | sed "s/kubernetes/${MC}/g" | base64 -w0)
kubectl -n tkg-system patch secret ${MC}-kubeconfig -p '{"data":{"value":"'"$encoded_data"'"}}' --type=merge
exit
exit

4. (jumpbox) Reload the kubeconfig

tanzu mc kubeconfig get ${MC} --admin

# Check the updated kubectl client cert
cat ~/.kube/config | yq ".users[]|select(.name==\"${MC}-admin\").user.client-certificate-data" | base64 -d | openssl x509 -noout -serial -dates

5. (jumpbox) Check the kubectl behavior

kubectl config use-context <MC_CONTEXT>
kubectl get nodes -owide

 

Update kubectl client-certificate for Workload Cluster

1. Set the Workload Cluster name

tanzu cluster list -A
CLUSTER=workload-test-1
NAMESPACE=default

# Check the kubectl client cert
cat ~/.kube/config | yq ".users[]|select(.name==\"${CLUSTER}-admin\").user.client-certificate-data" | base64 -d | openssl x509 -noout -serial -dates

2. Switch to the Management Cluster context

kubectl config use-context <MC_CONTEXT>

3. Delete the secret of the target cluster kuebconfig

kubectl -n ${NAMESPACE} delete secret ${CLUSTER}-kubeconfig

# If the previous command becomes stuck, remove the finalizer
kubectl -n ${NAMESPACE} patch secret ${CLUSTER}-kubeconfig --type=json -p='[{"op":"remove","path":"/metadata/finalizers/0"}]'

4. Regenerate a new secret forcibly

# Restart capi-controller-manager
kubectl -n capi-system rollout restart deployment/capi-controller-manager

# New secret will be recreated within 5 minutes
kubectl -n ${NAMESPACE} get secret ${CLUSTER}-kubeconfig

5. Reload the kubeconfig

tanzu -n ${NAMESPACE} cluster kubeconfig get ${CLUSTER} --admin

# Check the kubectl client cert
cat ~/.kube/config | yq ".users[]|select(.name==\"${CLUSTER}-admin\").user.client-certificate-data" | base64 -d | openssl x509 -noout -serial -dates

6. Check the kubectl behavior

kubectl config use-context <WORKLOAD_CONTEXT>
kubectl get nodes -owide

Additional Information

• KB - TKGm Certificate Rotation Guide