VCF 9.0 deployment fails at "Generate and Install VMCA Certificate on SDDC Manager" stage with error "Failed to install VMCA Certificate on SDDC Manager 127.0.0.1 Reference Token: <ID>"
search cancel

VCF 9.0 deployment fails at "Generate and Install VMCA Certificate on SDDC Manager" stage with error "Failed to install VMCA Certificate on SDDC Manager 127.0.0.1 Reference Token: <ID>"

book

Article ID: 423330

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • VCF deployment fails when trying to set SDDC Manager with VMCA signed certificate.

  • There are no issues with the misconfigured /etc/hosts file in the SDDC Manager where the SDDC Manager FQDN is associated with the Loopback IP Address.

  • Modifying the /etc/hosts file of the SDDC Manager to include both the vCenter server FQDN and IP doesn't help.

  • Per the SDDC Manager logs, the certificate chain presented by the vCenter server is in an incorrect format. The certificate chain is self-signed. The same is validated using the command "openssl s_client -connect <vCenter server FQDN>:443" on the SDDC Manager command line.

    [c.v.e.s.s.InstallSddcManagerVmcaCertificateLocalAction,dm-exec-5]  Installing SDDC Manager VCSA certificate
    ERROR [vcf_dm,69374f171f7a005b930d8e154b947cea,2a66] [c.v.e.s.s.InstallSddcManagerVmcaCertificateLocalAction,dm-exec-5]  API failure during install certificate Code: 500, error: {"errorCode":"CERT_REPLACEMENT_FAILED","arguments":[],"message":"Cannot replace existing certificate with the input cert. Validations did not pass.\nMake sure the input cert chain is valid. The structure must be:\n\u0027server cert\u0027 followed by \u0027intermediate certs\u0027 followed by \u0027CA cert\u0027\nOR\nA self signed server cert\nAll certs in the chain must conform to X.509 standards.\nAlso make sure that the DNS name in both the CN field and the optional
    Subject Alternative Name extension, is a resolvable hostname","causes":[{"type":"com.vmware.evo.sddc.appliance.utilities.error.CertValidatorException","message":"Cannot replace existing certificate with the input cert. Validations did not pass.\nMake sure the input cert chain is valid. The structure must be:\n\u0027server cert\u0027 followed by \u0027intermediate certs\u0027 followed by \u0027CA cert\u0027\nOR\nA self signed server cert\nAll certs in the chain must conform to X.509 standards.\nAlso make sure that the DNS name in both the CN field and the optional Subject Alternative Name extension, is a resolvable hostname"}],"referenceToken":"<Token-ID>"}
    com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException:
            at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.handleResponse(ApiClient.java:788)
            at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.execute(ApiClient.java:708)
            at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.execute(ApiClient.java:691)
            at com.vmware.cloud.foundation.rest.commonsvcs.service.CertificateServiceApi.installCertWithHttpInfo(CertificateServiceApi.java:1063)
            at com.vmware.cloud.foundation.rest.commonsvcs.service.CertificateServiceApi.installCert(CertificateServiceApi.java:1051)
            at com.vmware.evo.sddc.sddcmanager.InstallSddcManagerVmcaCertificateLocalAction.execute(InstallSddcManagerVmcaCertificateLocalAction.java:120)

  • Per operation manager logs of the SDDC Manager appliance, it is unable to construct a valid certificate chain. 

    Error checking certificate chain C=<CN>, CN=<CN>, SerialNumber=<SN> for validity.
    java.security.cert.CertificateException: Unable to construct a valid chain
            at org.bouncycastle.jsse.provider.ProvX509TrustManager.validateChain(ProvX509TrustManager.java:317)
            at org.bouncycastle.jsse.provider.ProvX509TrustManager.checkTrusted(ProvX509TrustManager.java:272)
            at org.bouncycastle.jsse.provider.ProvX509TrustManager.checkServerTrusted(ProvX509TrustManager.java:175)
            at org.bouncycastle.jsse.provider.ExportX509TrustManager_7.checkServerTrusted(ExportX509TrustManager_7.java:49)
            at com.vmware.vcf.secure.truststore.DynamicTrustManager.checkServerTrusted(DynamicTrustManager.java:52)
            at org.bouncycastle.jsse.provider.ImportX509TrustManager_5.checkServerTrusted(ImportX509TrustManager_5.java:71)
            at org.bouncycastle.jsse.provider.ProvSSLSocketWrap.checkServerTrusted(ProvSSLSocketWrap.java:127)
            at org.bouncycastle.jsse.provider.ProvTlsClient$1.notifyServerCertificate(ProvTlsClient.java:382)
            at org.bouncycastle.tls.TlsUtils.processServerCertificate(TlsUtils.java:4830)
            at org.bouncycastle.tls.TlsClientProtocol.handleServerCertificate(TlsClientProtocol.java:797)
            at org.bouncycastle.tls.TlsClientProtocol.receive13ServerCertificate(TlsClientProtocol.java:1596)
            at org.bouncycastle.tls.TlsClientProtocol.handle13HandshakeMessage(TlsClientProtocol.java:160)
            at org.bouncycastle.tls.TlsClientProtocol.handleHandshakeMessage(TlsClientProtocol.java:366)
            at org.bouncycastle.tls.TlsProtocol.processHandshakeQueue(TlsProtocol.java:715)
            at org.bouncycastle.tls.TlsProtocol.processRecord(TlsProtocol.java:591)
            at org.bouncycastle.tls.RecordStream.readRecord(RecordStream.java:247)
            at org.bouncycastle.tls.TlsProtocol.safeReadRecord(TlsProtocol.java:879)
            at org.bouncycastle.tls.TlsProtocol.blockForHandshake(TlsProtocol.java:427)
            at org.bouncycastle.tls.TlsClientProtocol.connect(TlsClientProtocol.java:88)
            at org.bouncycastle.jsse.provider.ProvSSLSocketWrap.startHandshake(ProvSSLSocketWrap.java:607)
            at org.bouncycastle.jsse.provider.ProvSSLSocketWrap.startHandshake(ProvSSLSocketWrap.java:583)
            at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
            at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
            at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
            at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
            at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
            at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
            at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
            at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
            at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)

  • Both the vCenter Server Appliance and SDDC Manager are sharing the same NTP source.

Environment

VMware Cloud Foundation 9.x

Cause

This issue occurs if the vCenter Server clock is ahead of the SDDC Manager.

In a completely new setup, the vCenter is just deployed by SDDC Manager and the same ensures that both appliances are properly synced with the same NTP server. In an existing setup, there is a time difference validation that tolerates up to 30 sec difference, however the issue is seen even if there is a time  difference by even a second or two where the vCenter Server clock is ahead of the SDDC Manager.

Resolution

Broadcom Engineering is working to address this issue in one of the upcoming releases. In the meantime, ensure that the there is no time synchronization issues between the vCenter Server and SDDC Manager.

Meanwhile, apply any one of the below set of workarounds.

Reboot the vCenter Server Appliance.

or

Change the tools-sync value to true and re-attempt the deployment. To do the same

  1. Export the final config.json file from the "Deploy and Validate" stage.
  2. Add a new parameter "time_tools_sync" if not present already and set the value to "True".
  3. Save the updated json file and re-import it to start with the deployment process.
Powered by Wolken Software