Import the vCenter root certificate into the SDDC manager TrustStore

Products

VMware Cloud Foundation

Issue/Introduction

This article shares the steps to remediate VCF environment in case of an Out Of Band certificate change. This is required in case the Root certificates for a WLD vCenter have been reset outside of SDDC Manager and the remediation involves importing the new certificate in SDDC Manager Trust store. Some of the scenarios where this is necessary are:

The SDDC Manager UI may display a blank screen with an error message similar to the following:

{"message":"Identity Internal Server Error","code":"IDENTITY_INTERNAL_SERVER_ERROR","status":500}
Exception traces similar to below may appear in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log on the SDDC Manager:

YYYY-MM-DDThh:mm:ss ERROR [vcf_om,########,####] [c.v.e.s.c.s.l.LicenseServiceFactory,om-exec-##] Failed to instantiate License Service with endpoint VC_FQDN
com.vmware.evo.sddc.common.services.psc.exception.PscException: Unable to obtain Security Token Service from SSO '<VC_FQDN>'
Exception traces similar to below may appear in /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log on the SDDC Manager:

YYYY-MM-DDTHH:MM:SS ERROR [common,########,####] [c.v.v.i.sync.utils.VcSyncManagerUtil,cs-exec-#] Error connecting to vCenter vcenter-#, with exception {} com.vmware.vim.vmomi.client.exception.SslException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:###) at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
...
YYYY-MM-DDTHH:MM:SS ERROR [common,########,####] Caused by: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found.
YYYY-MM-DDTHH:MM:SS ERROR [common,########,####] Renew VAPI session - failed to obtain new session
org.springframework.web.client.ResourceAccessException: I/0 error on POST request for "https://<VC_FQDN>/rest/com/vmware/cis/session": certificate_unknown (##)
On SDDC Manager UI, WLD vCenter certificate status has an alert with message "Connection cannot be established because certificate is not trusted" and upon selecting "review" option to attempt to trust the vCenter certificate, the certificate validation fails with error: "Failed to trust Certificate(s). Error while validating certificate"
Importing an existing workload domain with NSX using the VCF Operations fails at task:

Description: Generate NSX Input for configuring NSX
Progress Messages: Failed to fetch object name(s) from vCenter
Cause: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46) Unable to construct a valid chain No issuer certificate for certificate in certification path found.
Additionally in case of out of band VI WLD vCenter root certificate renewal, administrators logging into SDDC Manager may fail to see utilization details, commission hosts, add hosts into clusters.

Environment

VMware Cloud Foundation (VCF) 4.x
VMware Cloud Foundation (VCF) 5.x
VMware Cloud Foundation (VCF) 9.x

Cause

When certificate is replaced directly on the vCenter server managed by an SDDC Manager

The vCenter's own trust anchor changes.
SDDC Manager's internal trust store, which was configured to trust the previous certificate chain of that vCenter, becomes outdated and it can no longer securely communicate with or validate the identity of the WLD vCenter.

Resolution

Note:

Take a valid snapshot of the SDDC manager before executing the below commands.
The script process below is valid only when using self-signed certificates.

Scripted Process:

Upload the VcRootCaSync.py script to the SDDC Manager using Winscp or other file transfer protocol.
Run the script as the root user

python VcRootCaSync.pyExpected Output:
# python VcRootCaSync.py
Please provide SSO administrator user[[email protected]]:
Provide password for [email protected]:
Available vCenter Servers:
[1] ACTIVE | vcsa1.example.com
[2] ACTIVE | vcsa2.example.com
Select a vCenter server by entering the corresponding number: 1
Selected vCenter: vcsa1.example.com
Session token created successfully
Root certificate saved to /tmp/root.cer
vCenter vcsa1.example.com Root Certificate Found:
-----BEGIN CERTIFICATE-----
##################################################################
-----END CERTIFICATE-----
Using randomly generated Alias: abc-def-vcsa1_RootCrt_EA0G
vCenter Root certificate added to SDDC Manager trust stores.
Refreshing certificate store
Deleting root certificate from temp

Manual Process:

Obtain root certificate(s) from affected vCenter:
Depending on the certificate type, run the respective command to generate the root certificate(s) in /tmp on vCenter :
1. If vCenter is using self-signed certificate:
  /usr/lib/vmware-vmca/bin/certool --getrootca --cert=/tmp/root.cer
2. If vCenter is using custom root certificate(s), use the below script to fetch the root cert(s) :
  
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots | awk '
  BEGIN { cert_num=1; in_cert=0 }
  /-----BEGIN CERTIFICATE-----/ {
  in_cert=1
  outfile = "/tmp/root" cert_num ".cer"
  print $0 > outfile
  next
  }
  /-----END CERTIFICATE-----/ {
  print $0 > outfile
  close(outfile)
  in_cert=0
  cert_num++
  next
  }
  in_cert {
  print $0 > outfile
  }'
Add the certificates to SDDC Manager:
1. CLI Method using keytool command:
  1. Obtain the trusted certificates key by executing the following command:
    
    pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)
  2. Import the certificate into the trusted_certificates_store with the keytool.
    
    keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass
  3. Import the certificate into the cacerts store.
    
    keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeitNote: While the alias name can be arbitrary, it is recommended to use the alias name present in the vCenter's trusted root store for consistency.
  4. Verify the new certificate has been successfully added.
    
    keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass | less
  5. Refresh the trusted certificates using the API from the SDDC Manager.
    
    curl -X POST localhost/appliancemanager/trustedCertificates/refresh
2. UI Method using API Explorer (In case SDDC Manager UI is accessible):
  1. Convert the root certificate on the vCenter into single-line PEM format and copy the output.
    
    awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/vmca.crt
    For example:
    root@vcenter-1 [ ~ ]# awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/vmca.crt
    -----BEGIN CERTIFICATE-----
    <certificate..............>
    -----END CERTIFICATE-----\n
  2. Navigate to the API Explorer > APIs for managing Trusted Certificates > POST and input the certificate and certificateUsageType.
    Input the certificate in one line PEM format in the certificate field.
    Input TRUSTED_FOR_OUTBOUND into the certificateUsageType field.
  3. Click EXECUTE

Additional Information

Post importing the certificate, it might be necessary to restart the SDDC Manager services using below command:
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
To manually review the certificates on vCenter VECS store, refer to Manually reviewing certificates in VMware Endpoint Certificate Store for vSphere 6.x and 7.x
To fetch and import root certificates of other components like NSX, refer to SDDC Manager fails to trust integrated components certificate with error "Failed to trust Certificate(s). Error while validating certificate"

Attachments

VcRootCaSync get_app