VMware Cloud Director cell service does not start after reboot

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director cell service does not start after reboot
• Certificate was replaced recently
• In cell-runtime.log under /opt/vmware/vcloud-director/logs , we see errors that match one of the below:
  

  Key protection algorithm not found: java.security.KeyStoreException: Certificate chain is not valid

   OR

  Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
          javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity
   check failed
  
  Caused by: java.security.cert.CertificateExpiredException: NotAfter: [Expiration_date]

   
  
  

Environment

10.x

Cause

This happens if the certificate chain is out of order or expired

Resolution

• Cert chain should be in the following order : Server, Intermediate and then Root certificate.
• Take a backup of the .crt , .pem or ,pfx file in use and update the chain in the above order 
• Additionally, verify there are no spaces or unwanted characters between the certificates. 

  Certificate file's content should look like this:

  -----BEGIN CERTIFICATE-----
  xxxx
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  xxxx
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  xxxx
  -----END CERTIFICATE-----

Note: Since 10.5.1 and later, the Service Admin portal needs to be accessible as that is the only supported way to update cell certificates. In cases where the portal is not accessible any more, try one of the options below : 

• Option 1: Revert to the snapshots for the cells prior to the certificate update process and restart the process
• Option 2: Use API
  - Add cert library item :
  https://developer.broadcom.com/xapis/vmware-cloud-director-openapi/v37.1/certificate-library/index?scrollString=certificate%20library
  
  - Update endpoint :
  https://developer.broadcom.com/xapis/vmware-cloud-director-openapi/v38.1/cloudapi/1.0.0/cells/cellUrn/accessEndpoints/put/index?scrollString=cell%20certificate
• Option 3:
  Update the certificate_library_item_consumer table in vcloud database to point back to the old certificates. Contact Broadcom Support for more help. 

 

Additional Information

For more information, see SSL Certificate Creation and Management of Your VMware Cloud Director Appliance.