"Health-Check operations for SDDC" task fails in VCF SDDC manager due to certificate verification failure

Products

• VMware SDDC Manager

Issue/Introduction

• A few checks including ntp-check, connectivity-check and password-check fail during health check causing overall failure of "Health-Check operations for SDDC" task
• Below logs are observed in /var/log/vmware/vcf/sddc-support/vcf-sos.log indicating certificate verification failure when connecting to certain components:
  
  Loading CA CERTS
HTTPSConnectionPool(host='<hostname>', port=443): Max retries exceeded with url: /api/session (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate signature failure (_ssl.c:1006)')))
  
  
• All certificates of VCF instances are signed by openssl CA
• The issue started after the latest certificate replacement
• Checking the validity of the signed certificate with below command fails with "verification error". The command should return "OK" if the certificate is valid.
  
  openssl verify -CAfile <openssl_ca_cert> <signed_cert>

Environment

VMware SDDC Manager 9.x

Cause

The openssl CA certificate stored in SDDC manager trusted certificate store is corrupted or invalid.

Resolution

Re-import a good copy of openssl CA certificate to SDDC manager trusted certificate store.

If the openssl CA certificate kept in vCenter trusted root store appears valid, below procedure can be followed to import the openssl CA certificate from vCenter to SDDC manager: How to import the vCenter root certificate into the SDDC manager TrustStore

Additional Information

The trusted certificates of vCenter and SDDC manager can be gathered using below commands respectively:

vCenter:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots

SDDC manager: 

keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store --storepass <trust store key>
keytool -list -v -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

Below commands can be used to import certificate to SDDC manager trusted certificate store:

keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store --storepass <trust store key>

keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

Reference: Keytool Commands for interacting with certificates in SDDC Manager Java Keystores