The VMware Cloud Foundation (VCF) installation failed at the vCenter certificate installation stage, showing the error: "Failed to install certificates on VCF Installer."
search cancel

The VMware Cloud Foundation (VCF) installation failed at the vCenter certificate installation stage, showing the error: "Failed to install certificates on VCF Installer."

book

Article ID: 409985

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • VCF installation failed at Validate & Deploy with error: "Failed to install certificates on VCF installer."

  • On checking domain manager logs in the location - /var/log/vmware/vcf/domainmanager/domainmanager.log we get below error:
DEBUG [vcf_dm,#########dffb96c306a1e#################] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-2]  Contract task Install Certificates on Installer Appliance input: {"remoteEndpoints":[],"aliasToCertificateM
ap":{},"installRemoteEndpointsCertificates":true,"brownfieldProductsThumbprints":{"VC-FQDN":"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##"},"_executionUuid":"92b####-7640-
####-91ca-f7#####e743"}
WARN  [vcf_dm,68c######b96c306a1e#######a8,####] [c.v.v.s.config.TrustAllTrustManager,dm-exec-2]  Trusting server
WARN  [vcf_dm,68c######b96c306a1e#######a8,####] [c.v.v.s.config.TrustAllTrustManager,dm-exec-2]  Trusting server
[vcf_dm,68c######b96c306a1e#######a8,####] [c.v.e.s.v.a.InstallCertificatesOnInstallerAction,dm-exec-2]  Trusting certificate for existing product: VC-FQDN
[vcf_dm,68c######b96c306a1e#######a8,####] [c.v.e.s.v.a.InstallCertificatesOnInstallerAction,dm-exec-2]  Failed to install certificates on VCF Installer appliance
com.vmware.vcf.rest.api.runtime.ApiException:
        at com.vmware.vcf.rest.api.runtime.ApiClient.handleResponse(ApiClient.java:788)
        at com.vmware.vcf.rest.api.runtime.ApiClient.execute(ApiClient.java:708)
        at com.vmware.vcf.rest.api.service.TrustedCertificatesApi.addTrustedCertificateWithHttpInfo(TrustedCertificatesApi.java:141)
        at com.vmware.vcf.rest.api.service.TrustedCertificatesApi.addTrustedCertificate(TrustedCertificatesApi.java:127)
        at com.vmware.evo.sddc.validation.action.InstallCertificatesOnInstallerAction.populateBrownfieldProductsTrust(InstallCertificatesOnInstallerAction.java:157)
        at com.vmware.evo.sddc.validation.action.InstallCertificatesOnInstallerAction.execute(InstallCertificatesOnInstallerAction.java:82)
        at com.vmware.evo.sddc.validation.action.InstallCertificatesOnInstallerAction.execute(InstallCertificatesOnInstallerAction.java:63)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:403)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:517)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)
        at jdk.internal.reflect.GeneratedMethodAccessor370.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
  • On checking commonsvcs logs in the location - /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log we got the below entries:

    Caused by: com.vmware.evo.sddc.appliance.utilities.error.ApplianceException: Error while validating certificate
        at com.vmware.evo.sddc.appliance.utilities.CertificateUtilImpl.convertToX509CertificateChain(CertificateUtilImpl.java:275)
        at com.vmware.evo.sddc.appliance.utilities.CertificateUtilImpl.updateCertSpecsIfRequired(CertificateUtilImpl.java:396)
        at com.vmware.evo.sddc.appliance.utilities.CertificateUtilImpl.updateSddcManagerTrustStores(CertificateUtilImpl.java:173)
        at com.vmware.evo.sddc.appliance.utilities.v1.CertificateUtilityImpl.addTrustedCertificate(CertificateUtilityImpl.java:44)
        at com.vmware.evo.sddc.appliance.utilities.api.rest.v1.CertificateController.addTrustedCertificate(CertificateController.java:49)
        ... 151 common frames omitted
Caused by: java.security.cert.CertificateExpiredException: NotAfter: DD MM HH:MM:SS
        at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277)
        at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:621)
        at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:594)
        at com.vmware.evo.sddc.appliance.utilities.CertificateUtilImpl.convertToX509CertificateChain(CertificateUtilImpl.java:269)
        ... 155 common frames omitted
INFO  [common,68c####06d8db####aa4f3f8#######9b] [c.v.v.l.a.a.ActivityLoggingInterceptor,http-nio-127.0.0.1-7100-exec-2] {"username":"domainmanager","timestamp""clientIP":"127.0.0.1","userAgent":"Swagger-Codegen/1.0.0/java","api":"/v1/sddc-manager/trusted-certificates","httpMethod":"POST","httpStatus":500,"operation":"Add a trusted certificate to the appliance's trust store","remoteIP":"127.0.0.1","duration":98}

Environment

VMware Cloud Foundation 9.x

Cause

VCF installation fails during certificate installation due to expired trusted root certificates present in the vCenter and VCF Installer's trust store, triggering a validation failure.

Resolution

1. Remove the expired certificate from the vCenter trusted root store following the KB: Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS).

2. Verify if expired certificates are present in the SDDC Manager's trusted certificate store and remove them if found.

  • SSH to SDDC Manager with vcf user and su to root.
  • Retrieve the trust store password: 
    KEY=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)
  • List certificates in the trust store: 
    keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $KEY
  • Delete expired certificates by alias: 
    keytool -delete -alias <aliasname> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $KEY

3. Retry the VCF Installation.

Powered by Wolken Software