Error encountered while updating the vCenter certificate or credentials for a Management Cluster

Products

• VMware Telco Cloud Automation

Issue/Introduction

• When attempting to update the vCenter certificate or credentials for a Management Cluster in the TCA-CP UI, the operation fails.
  
  
• Error in TCA-CP UI during credential update
  "failed to login management cluster <xxx-tkg-mgmt-01> when updating credential of vc, err: failed to login management cluster xxx-tkg-mgmt-01"
  
  
• Trust Error on TCA-CP UI
  "Re-establish Trust - Operation Failed. Please check details."
  
  
• The environment (TCA_M & CP) was recently restored from backup
  
  
• The ''UpdateVcThumbprintJob" shows as "failed" status in the Tasks and Events view under CaaS Infrastructure
  
  
• TCA_M connection endpoints to vCenter on port 9443 are confirmed to be reachable
  
  
• The vCenter certificate SHA1 and SHA256 thumbprints have been validated manually using 'openssl' and match the thumbprints on the TCA-CP

Environment

3.4

Cause

This issue occurs when the 'k8s-bootstrapper' pod has been restarted and no PUT or POST actions have been performed against it since the restart. This state prevents the successful update of vCenter certificates or credentials on the management cluster.

Resolution

To resolve this issue, perform the following workaround steps to manually refresh the kubeconfig and context for the management cluster.

1. SSH into the TCA-CP node with root or sudo privileges and execute the following command to access the shell of the `k8s-bootstrapper` pod:

# kubectl -n tca-cp-cn exec -ti $(kubectl -n tca-cp-cn get pods -l app=k8s-bootstrapper -o jsonpath='{.items[0].metadata.name}') -- bash

2. Replace `<mc_name>` with the actual name of your Management Cluster and run the following command to export the name and fetch the kubeconfig:

# export MC_NAME="<mc_name>" && curl http://localhost:8888/api/v1/managementcluster/$(curl -s http://localhost:8888/api/v1/managementclusters | jq -r --arg mc ${MC_NAME} '.[] | select(.clusterName=$mc) | .id')/kubeconfig > /tmp/kubeconfig

3. Merge the retrieved Management Cluster kubeconfig with the default Tanzu and kubectl configuration files by running the following block of commands:

cd /opt/vmware/kbs/kbs-tkg254/tanzu/ && KUBECONFIG=.kube/config:/tmp/kubeconfig kubectl config view --merge --flatten > merged_kubeconfig && cp merged_kubeconfig .kube/config && rm -f merged_kubeconfig

cd /opt/vmware/kbs/kbs-tkg254/tanzu/ && KUBECONFIG=.kube-tkg/config:/tmp/kubeconfig kubectl config view --merge --flatten > merged_kubeconfig && mkdir -p .kube-tkg && cp merged_kubeconfig .kube-tkg/config && rm -f merged_kubeconfig

4. Re-add the context for the Management Cluster to the Tanzu CLI. Ensure `<mc_name>` is replaced with your Management Cluster name:

# export MC_NAME="<mc_name>" && HOME=/opt/vmware/kbs/kbs-tkg254/tanzu/ /opt/vmware/kbs/kbs-tkg254/bin/tanzu context create ${MC_NAME} --kubeconfig /opt/vmware/kbs/kbs-tkg254/tanzu/.kube-tkg/config --kubecontext ${MC_NAME}-admin@${MC_NAME}



5. Return to the TCA-CP UI and validate the vCenter certificate or credential update operation