Error: "Error in TLS server certificate thumbprint check: certificate thumbprints are not matched, expected" after updating endpoint certificate via UI

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

• Manually updated vCenter endpoint certificate via TCA UI

  • Using vSphere SSO/AD credentials

  • Updated from Administration > Connected Endpoints > Click ... (edit) on vCenter Server endpoint > Update Certificate

• Error received after certificate update:
  
Certificate of vCenter is untrusted. Please click "Update Certificate" to provide the latest vCenter CA certificate in TCA.

• Endpoint Details report: 

status:
  conditions:
    - type: Ready
      reason: ThumbprintMismatch
      status: 'False'
      message: >-
        [TLS-Probe] Error in TLS server certificate thumbprint check:
        certificate thumbprints are not matched, expected:
        ######################D4, actual:
        ######################20

Environment

TCA: 3.3, 3.4

TCP: 5.0.1, 5.1

Cause

When a vCenter endpoint has multiple certificates in its chain, the UI picks up the root certificate SHA-1 thumbprint and populates it as a SHA-1 thumbprint for the vCenter certificate update input instead of picking the leaf certificate SHA-1 thumbprint. 

Resolution

Resolved in the 3.3.0.1 patch

Workaround

1. Capture the API Request

   1. Log in to the TCA Manager UI.

   2. Press F12 (or Cmd+Option+I on Mac) to open Developer Tools and go to the Network tab.

   3. Go to the endpoint in question and trigger the Certificate Update operation in the UI.

   4. Once the certificate update fails, go to the Network tab and search forcertImport.

   5. Right-click the failed certImport network entry and select Copy > Copy as cURL(POSIX)

2. Modify the Payload

   1. Paste the copied curl command into a text editor.

   2. Update the thumbprint field within the --data-raw section with the leaf certificate thumbprint of your new certificate.
      
      Replace:
      "thumbprint":"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##"
      
      Obtain thumbprint:
      openssl x509 -in <leafCertificate>.crt -noout -fingerprint
      
      or you can find the thumbprint using the below command:
      echo | openssl s_client -connect <vCenter FQDN>:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1
      
      

3. Execute the Manual Update

   1. Copy the entire modified curl command and add --insecure in the end as given in the sample Payload example

   2. Open a terminal, paste the curl command, and press Enter.
      
      Note: The session expired in few seconds, so you need to run the copied curl command quickly by modifying it.
      

Additional Information

Sample Payload: 

curl 'https:/<tca-m-IP>/tca/monitoring/api/v1/endpoints/<endpointId>/action/certImport' \
  -H 'accept: application/json' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -b 'JSESSIONID=EF89D3D31945A32A8A44A344F5A19816; NSPSESSIONID=2ea9e412:01d0:4053:821c:c3bced3deeff' \
  -H 'origin: https://#.#.#.#' \
  -H 'pragma: no-cache' \
  -H 'priority: u=1, i' \
  -H 'referer: https://#.#.#.#/telco/ui/tca-manager/' \
  -H 'sec-ch-ua: "Chromium";v="146", "Not-A.Brand";v="24", "Microsoft Edge";v="146"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0' \
  --data-raw '{"systemUuids":["############-########-####-####-####-########","############-########-####-####-####-########"],"certificate":"-----BEGIN CERTIFICATE-----\n#########################################################\n-----END CERTIFICATE-----","thumbprint":"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##"}' --insecure

Sample Response:

{
  "status" : "QUEUED",
  "intentId" : "########-####-####-####-############",
  "message" : "Certificate update operation is in progress",
  "actionType" : "UPDATE_CERTIFICATE"
}%    

In some cases the vsphere-csi daemon pods go into CrashLoopBackOff state after vCenter thumbprint is updated. In that case, verify the thumbprint from the secrets on the clusters and restart the csi pod if the thumprint are updated successfully:

Login to the workload clusters

kubectl get secret vsphere-config-secret -n kube-system -o jsonpath='{.data.vsphere\.conf}' | base64 --decode

Perform restart the vsphere-csi-controller pod by running below command on the workload clusters:

kubectl rollout restart deploy/vsphere-csi-controller -n vmware-system-csi