Error encountered while updating the vCenter certificate or credentials for a Management Cluster "failed to login management cluster <xxx-tkg-mgmt-01> when updating credential of vc, err: failed to login management cluster xxx-tkg-mgmt-01"

Products

• VMware Telco Cloud Automation

Issue/Introduction

• TCA_M & CP was restored on the environment
• TCA_M connection endpoints to vCenter looks good on port 9443
• UpdateVcThumbprintJob is in failed state on Tasks and Events under Caas Infrastructure
• Validated the vCenter certificate sha1 & sha26 thumbprints with the below commands and it matches with thumbprint on the TCA-CP.
  openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1
openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256 
  
  
• Found trust errors on the TCA_CP UI, "Re-establish Trust - Operation Failed. Please check details."

Environment

3.4

Cause

vCenter certificate/Credentials update fails on the management cluster if the k8s-bootstrapper pod was restarted and no PUT/POST actions were performed since restart

Resolution

Please follow the below workaround steps:

Below is the workaround:
1 . SSH into the TCA-CP node and access the k8s-bootstrapper pod.
kubectl -n tca-cp-cn exec -ti $(kubectl -n tca-cp-cn get pods -l app=k8s-bootstrapper -o jsonpath='{.items[0].metadata.name}') -- bash
2. Get the kubeconfig for the Management Cluster.
Replace <mc_name> with your actual Management Cluster name.
export MC_NAME="<mc_name>" && curl http://localhost:8888/api/v1/managementcluster/$(curl -s http://localhost:8888/api/v1/managementclusters | jq -r --arg mc ${MC_NAME} '.[] | select(.clusterName=$mc) | .id')/kubeconfig > /tmp/kubeconfig
3. Merge the Management Cluster kubeconfig with the default Tanzu and kubectl 
cd /opt/vmware/kbs/kbs-tkg254/tanzu/ && KUBECONFIG=.kube/config:/tmp/kubeconfig kubectl config view --merge --flatten > merged_kubeconfig && cp merged_kubeconfig .kube/config && rm -f merged_kubeconfig 
cd /opt/vmware/kbs/kbs-tkg254/tanzu/ && KUBECONFIG=.kube-tkg/config:/tmp/kubeconfig kubectl config view --merge --flatten > merged_kubeconfig && mkdir -p .kube-tkg && cp merged_kubeconfig .kube-tkg/config && rm -f merged_kubeconfig
4. Add the Tanzu CLI context for the Management Cluster:
export MC_NAME="<mc_name>" && HOME=/opt/vmware/kbs/kbs-tkg254/tanzu/ /opt/vmware/kbs/kbs-tkg254/bin/tanzu context create ${MC_NAME} --kubeconfig /opt/vmware/kbs/kbs-tkg254/tanzu/.kube-tkg/config --kubecontext ${MC_NAME}-admin@${MC_NAME}

Additional Information

Refer TCA 3.4 Release notes

Issue 3554436: Error encountered while updating the vCenter certificate or credentials for a Management Cluster.