SDDC Manager certificate replacement failed with an API exception error stating "com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException"
search cancel

SDDC Manager certificate replacement failed with an API exception error stating "com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException"

book

Article ID: 395846

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • SDDC Manager certificate installation failed during validation -  'Throwing error from session middleware -getaddrinfo ENOTFOUND localhost'

  • Certificate chain is valid and the DNS name in both the CN field and the optional Subject Alternative Name extension, is a resolvable hostname.
  • SDDC Manager certificate replacement fails with a com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException.
  • This error occurs because the commonsvcs API, called during the replacement process, validates the FQDN resolution of the input certificate.
  • Details of this exception can be found in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log - 

YYYY-MM-DDTHH:MM:SS.Z ERROR [vcf_om,<id>] [c.v.v.c.s.SddcManagerCertificatePluginService,om-exec-17] SDDC Manager Certificate Replacement failed:
com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException:
atcom.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.handleResponse(ApiClient.java:788)

java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840) 

YYYY-MM-DDTHH:MM:SS.Z ERROR [vcf_om,<id>][c.v.v.c.s.SddcManagerCertificatePlugin,om-exec-22] SDDC Manager Certificate Replacement failed: java.net.ConnectException: Failed to connect to localhost/[#:#:#:#:#:#:#:1]:7100  <--IPv6
com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException: java.net.ConnectException: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:7100
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

  • Upon reviewing the /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log, the following entries were observed - 

YYYY-MM-DDTHH:MM:SS.Z ERROR [common,687fad02abcc47f9b679e918ffe47198,8c71] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-319] [1BH###] CERT_REPLACEMENT_FAILED Cannot replace existing certificate with the input cert. Validations did not pass.
Make sure the input cert chain is valid. The structure must be:
server cert followed by intermediate certs followed by CA cert
OR
A self signed server cert
All certs in the chain must conform to X.509 standards.
Also make sure that the DNS name in both the CN field and the optional Subject Alternative Name extension, is a resolvable hostname
com.vmware.evo.sddc.appliance.utilities.error.ApplianceManagerException: Cannot replace existing certificate with the input cert. Validations did not pass.
Make sure the input cert chain is valid. The structure must be:
server cert followed by intermediate certs followed by CA cert
OR
A self signed server cert
All certs in the chain must conform to X.509 standards.
Also make sure that the DNS name in both the CN field and the optional Subject Alternative Name extension, is a resolvable hostname
        at com.vmware.evo.sddc.appliance.utilities.api.rest.CertificateController.installCert(CertificateController.java:167)
atalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.vmware.evo.sddc.appliance.utilities.error.CertValidatorException: Error while validating certificate
        at com.vmware.evo.sddc.appliance.utilities.NginxCertUtilityImpl.installCert(NginxCertUtilityImpl.java:173)
        at com.vmware.evo.sddc.appliance.utilities.api.rest.CertificateController.installCert(CertificateController.java:165)
        ... 139 common frames omitted
Caused by: java.security.cert.CertificateException: Hostname in CN field [SDDC_FQDN] could not be resolved to an IP address of the SDDC manager [#.#.#.#]
        at com.vmware.evo.sddc.appliance.utilities.utils.SslCertValidator.validateCNAndSANDnsName(SslCertValidator.java:295)
        at com.vmware.evo.sddc.appliance.utilities.utils.SslCertValidator.validateCertChain(SslCertValidator.java:262)
        ... 144 common frames omitted

Environment

  • VMware SDDC Manager 5.2.x

Cause

  • This error occurs because the commonsvcs API, called during the replacement process, validates the FQDN resolution of the input certificate.
  • This validation is failing because a recent security update upgraded the DNS Java library from version 2.1.9 to 3.6.3.
  • This upgrade introduces significant API changes and incompatibilities, leading to DNS resolution issues.

Resolution

  • Important Note - Before you start, take a snapshot of your SDDC Manager VM.
  1. SSH into the SDDC Manager using root user credentials.
  2. Back up the hosts file: Run the command - cp /etc/hosts /etc/hosts.bak.
  3. Edit the /etc/hosts file. Find and comment out the lines that map the SDDC Manager's Fully Qualified Domain Name (FQDN) to 127.0.0.1 and ::1 (Both the IPv4 and IPv6 localhost entries).
  4. Add the /etc/hosts file with the entries using the shortname instead of FQDN:

127.0.0.1 SDDC_Manager_Short_Name localhost
::1 SDDC_Manager_Short_Name localhost ipv6-localhost ipv6-loopback

  1. Retry the certificate replacement workflow that previously failed.
  2. Once the workflow finishes successfully, revert the changes you made in step 3 by uncommenting the original entries in the /etc/hosts file. This will restore the FQDN resolution mechanism to its default state for future operations.
Powered by Wolken Software