Update vCenter certificate for TKG cluster

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

All clusters, nodepools, and Network Function (NF) operations require a secure trust to vCenter via a vSphere certificate thumbprint. 

Symptoms:

• Workload clusters display the following error when checking their endpoints:
  Error in TLS server certificate thumbprint check: certificate thumbprints are not matched

• Endpoints status for vCenter in Unhealthy in the GUI.

Environment

TCA: 3.0, 3.1, 3.2

TCP: 3.0, 3.1, 5.0

Cause

If a vCenter's certificate has been updated, the vCenter certificate and thumbprint must be updated to restore functionality

Resolution

Resolved in TCA 3.3. See Updating Certificate/user Credentials of vCenter on TCA and Linked CaaS Clusters

Workaround:

1. Re-Import the vCenter Certificate to TCA-M and TCA-CP

• Log in to the TCA Appliance Manager 
  <tca-m/tca-cp-ip>:9443.
  
  
• Click Certificate > Trusted CA Certificate > IMPORT.
  
  
• Select the trusted certificate type that you want to import and do one of the following:

• • • Browse and select the file to import.
    • Type the URL of the certificate.
    • Paste the certificate file content.
      
      
• Click Apply.

     2. Update the vCenter thumbprint in the TKG Clusters 

• SSH into the TCA-CP that controls the management clusters using the admin credentials.
  ssh admin@<tca-cp-ip>
  
  
• Check connectivity to retrieve the update-vc-tp script.
  
  curl -kfsSL https://packages.broadcom.com/artifactory/tca-distro/kb/vc-updater/tca3.0/update-vc-tp.sh | bash -s -- -h

NOTE: For airgap environments, users should download the script to another location first, then copy it to the TCA-CP.

• Run the script with the FQDN as stored in vcenterprime server.address field (ssh to management node, 'kubectl get vcenterprime <name p-xxx> -n tca-system -o yaml'  ) 
  
  curl -kfsSL https://packages.broadcom.com/artifactory/tca-distro/kb/vc-updater/tca3.0/update-vc-tp.sh | bash -s -- -d <vCenter-fqdn>

  NOTE: In case of different vCenter for management clusters and for Workload clusters, execute the above command by giving the both vcenter (first execute the command with the VC address responsible for MC and then the same command with WC vCenter FQDN)
  
  
• Navigate to Connected Endpoints, find the corresponding vCenter, and confirm the status of the vCenter has been modified.

• Click and dismiss the message to acknowledge the vCenter certificate change.
  
  
• In TCA-M UI, Validate the vCenter credentials in the virtual infrastructure and save the changes.

1.  

Additional Information

In some cases the script may report vSphere CSI Secret Thumbprint mismatch for workload clusters. In this case vSphere-csi daemon set pods go into CrashLoopBackOff state after vCenter thumbprint is updated. Or, the vsphere-csi pods could take several minutes to be recreated.

Cluster Name                      : worload-cluster1
        Cluster Type              : classy
        TKG VSphere CSI Secret    :
                Thumbprint        : <REDACTED>   Mismatched.
        TKG VSphere CPI Secret    :
                Thumbprint        : <REDACTED>   Mismatched.

Workaround:

For clusters with node pools, recreate the vsphere-csi package installation by running below commands on the workload clusters:

# For classy clusters kubectl delete pkgi -n tkg-system <cluster-name>-vsphere-csi

# For standard clusters kubectl delete pkgi -n tkg-system vsphere-csi

For clusters without node pools, restart the vsphere-csi-controller pod by running below command on the workload clusters:

kubectl rollout restart deploy/vsphere-csi-controller -n vmware-system-csi

Note: This workaround is added to TCA 3.3

If even after applying the KB, the workload clusters still shows the status as Unhealthy in the GUI for the vCenter endpoint, then verify from the workload clusters secrets if the thumprint is updated or not.
Login to the workload clusters

kubectl get secret vsphere-config-secret -n kube-system -o jsonpath='{.data.vsphere\.conf}' | base64 --decode

Verify endpoint status is true or not 

kubectl get endpoints.m -A

If the status is false from the CLI as well, then machines must be in not ready state which prevent the controller from updating the underlying infrastructure secrets. Identify and troubleshoot the underlying cause of the NotReady status on the affected workload nodes to restore them to a fully operational and responsive state. Refer KB 378405 to troubleshoot on node not ready status