Import the vCenter root certificate into the SDDC manager TrustStore

Products

• VMware Cloud Foundation

Issue/Introduction

• If the vCenter root certificate is not published to the SDDC Manager truststores, the following errors may occur:
  
  
  • Unable to see utilization details in the SDDC UI.
  • Unable to commission hosts.
  • Unable to add hosts into clusters.
  • Unable to log in to the SDDC UI.
    
    
• Exception traces may appear in the commonvsvc log or the operationsmanager log within the SDDC Manager:
  
  
  • /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log

YYYY-MM-DDTHH:MM:SS ERROR [common,#################] [c.v.v.i.sync.utils.VcSyncManagerUtil,cs-exec-#] Error connecting to vCenter vcenter-#, with exception {} com.vmware.vim.vmomi.client.exception.SslException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:###) at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
  
  
  • /var/log/vmware/vcf/operationsmanager/operationsmanager.log

YYYY-MM-DDTHH:MM:SS. +0000 ERROR [vcf_om,#################] [c.v.e.s.c.s.l.LicenseServiceFactory,om-exec-##] Failed to instantiate License Service with endpoint VC_FQDN
com.vmware.evo.sddc.common.services.psc.exception.PscException: Unable to obtain Security Token Service from SSO 'VC_FQDN'
    
    
• The SDDC Manager GUI may display a blank screen with an error message similar to the following:
  
  {"message":"500 - \"{\\\"errorCode\\\":\\\"IDENTITY_INTERNAL_SERVER_ERROR\\\",\\\"arguments\\\":[],\\\"message\\\":\\\"Identity Internal Server Error\\\",\\\"referenceToken\\\":\\\"ABC###\\\"}\""}
  
  
• In a VCF 9.0 environment, logging into the UI displays an 'Internal Server Error'.
  
  
• The traces of exception in the /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log is as below:
  
  Caused by: java. security.cert. CertPathBuilderException: No issuer certificate for certificate in certification path found.
Renew VAPI session - failed to obtain new session
org. springframework.web. client. ResourceAccessException: I/0 error on POST request for "https://VC_FQDN/rest/com/vmware/cis/session": certificate_unknown (##)
at org. springframework.web. client. RestTemplate. createResourceAccessException (RestTemplate. java : ###)
at org. springframework.web. client. RestTemplate. doExecute (RestTemplate. java : ###)
  
  
• Importing an existing workload domain with NSX using the VCF Operations fails at task:
  
  Description: Generate NSX Input for configuring NSX
Progress Messages: Failed to fetch object name(s) from vCenter
Cause: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46) Unable to construct a valid chain No issuer certificate for certificate in certification path found.

Environment

• VMware Cloud Foundation (VCF) 4.x
• VMware Cloud Foundation (VCF) 5.x
• VMware Cloud Foundation (VCF) 9.x

Cause

The vCenter root certificate has been changed out-of-band and the SDDC Manager is still referencing the old root certificate. This can occur when option 8 is run in the certificate manager utility on vCenter.

Resolution

Note:

• Take a valid snapshot of the SDDC manager before executing the below commands.
• The script process below is valid only when using self-signed certificates.

Scripted Process:

1. Upload the VcRootCaSync.py script to the SDDC Manager using Winscp or other file transfer protocol.
   
   
2. Run the script as the root user
   
   python VcRootCaSync.py

Expected Output:
   

   /home/vcf ]# python VcRootCaSync.py
Please provide SSO administrator user[[email protected]]:
Provide password for [email protected]:
Available vCenter Servers:
[1] ACTIVE | vcsa1.example.com
[2] ACTIVE | vcsa2.example.com

           Select a vCenter server by entering the corresponding number: 1
        Selected vCenter: vcsa1.example.com

            Session token created successfully
         Root certificate saved to /tmp/root.cer

   vCenter vcsa1.example.com Root Certificate Found:
-----BEGIN CERTIFICATE-----
##################################################################

   -----END CERTIFICATE-----

            Using randomly generated Alias: abc-def-vcsa1_RootCrt_EA0G

            vCenter Root certificate added to SDDC Manager trust stores.
         Refreshing certificate store
         Deleting root certificate from temp
   
   

Manual Process:

1. SSH into the vCenter server and get the root certificate.
   
   
2. If using the default certificate, run the following command:
   
   /usr/lib/vmware-vmca/bin/certool --getrootca --cert=/tmp/root.cer
   
   
3. If using a custom root certificate:
   
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots
   
   
4. Copy the root certificate to the temp directory on the SDDC Manager.
   

   vi /tmp/root.cer to create an empty file with vi.

   Press i to enter insert mode.

   Paste the body of the certificate into the root.cer file.

   Press esc to exit insert mode.

   Type wq! and hit enter to write changes to the file and exit the vi editor.

5. Obtain the trusted certificates key by issuing the following command:
   
   pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)
   
   
6. Import the certificate into the trusted_certificates_store with the keytool.
   
   keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass <trust store key>
   
   Example:
   
   keytool -importcert -alias new_mgmt_root -file /tmp/root.cer -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass
   
   
7. Import the certificate into the cacerts store.
   
   keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit
   
   Example:
   
   keytool -importcert -alias new_mgmt_root -file /tmp/root.cer -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

Note: While the alias name can be arbitrary, it is recommended to use the alias name present in the vCenter's trusted root store for consistency.
   
   
8. Verify the new certificate has been successfully added.
   
   keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass | less
   
   
9. Refresh the trusted certificates using the API from the SDDC Manager.
   
   curl -X POST localhost/appliancemanager/trustedCertificates/refresh

Workaround:

If the SDDC UI is accessible, the root certificate can also be imported from the API Explorer.

1. Convert the root certificate on the vCenter into single-line PEM format and copy the output.
   
   awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/vmca.crt
   

   For example:
   
   root@vcenter-1 [ ~ ]# awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/vmca.crt
-----BEGIN CERTIFICATE-----
<certicate..............>
-----END CERTIFICATE-----\n

2. Navigate to the API Explorer in the SDDC UI and input the certificate and certificateUsageType.
   
   API Explorer > APIs for managing Trusted Certificates > POST
   
   
   • Input the certificate in one line PEM format in the certificate field.
   • Input TRUSTED_FOR_OUTBOUND into the certificateUsageType field.

   

3. Click Execute.

Additional Information

Manually reviewing certificates in VMware Endpoint Certificate Store for vSphere 6.x and 7.x

Uploading a file to an appliance using Winscp