SDDC Manager certificate replacement failed with an API exception error stating "com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException"

Products

• VMware SDDC Manager

Issue/Introduction

• SDDC Manager certificate replacement fails with a com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException.
• This error occurs because the commonsvcs API, called during the replacement process, validates the FQDN resolution of the input certificate.
• Details of this exception can be found in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log - 

YYYY-MM-DDTHH:MM:SS.Z ERROR [vcf_om,<id>] [c.v.v.c.s.SddcManagerCertificatePluginService,om-exec-17] SDDC Manager Certificate Replacement failed:
com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException:
atcom.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.handleResponse(ApiClient.java:788)
at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.execute(ApiClient.java:708)
at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.execute(ApiClient.java:691)
at com.vmware.cloud.foundation.rest.commonsvcs.service.CertificateServiceApi.installCertWithHttpInfo(CertificateServiceApi.java:943)
at com.vmware.cloud.foundation.rest.commonsvcs.service.CertificateServiceApi.installCert(CertificateServiceApi.java:931)
at com.vmware.vcf.certmgmt.sddcmgr.SddcManagerCertificatePluginService.replaceCertificate(SddcManagerCertificatePluginService.java:166)
at com.vmware.vcf.certmgmt.sddcmgr.SddcManagerCertificatePlugin.replaceCertificate(SddcManagerCertificatePlugin.java:105)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703)
at com.vmware.vcf.certmgmt.sddcmgr.SddcManagerCertificatePlugin$$SpringCGLIB$$0.replaceCertificate(<generated>)
at com.vmware.vcf.certmgmt.service.orch.impl.CertificateOperationOrchestratorImpl.replaceCertificate(CertificateOperationOrchestratorImpl.java:1371)
at com.vmware.vcf.certmgmt.service.orch.impl.CertificateOperationOrchestratorImpl.lambda$createCertificateOperationTask$32(CertificateOperationOrchestratorImpl.java:541)
at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840) 

YYYY-MM-DDTHH:MM:SS.Z ERROR [vcf_om,<id>][c.v.v.c.s.SddcManagerCertificatePlugin,om-exec-22] SDDC Manager Certificate Replacement failed: java.net.ConnectException: Failed to connect to localhost/[#:#:#:#:#:#:#:1]:7100  <--IPv6
com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException: java.net.ConnectException: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:7100
        at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.execute(ApiClient.java:711)
        at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.execute(ApiClient.java:691)
        at com.vmware.cloud.foundation.rest.commonsvcs.service.CertificateServiceApi.installCertWithHttpInfo(CertificateServiceApi.java:943)
        at com.vmware.cloud.foundation.rest.commonsvcs.service.CertificateServiceApi.installCert(CertificateServiceApi.java:931)
        at com.vmware.vcf.certmgmt.sddcmgr.SddcManagerCertificatePluginService.replaceCertificate(SddcManagerCertificatePluginService.java:166)
        at com.vmware.vcf.certmgmt.sddcmgr.SddcManagerCertificatePlugin.replaceCertificate(SddcManagerCertificatePlugin.java:105)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
        at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703)
        at com.vmware.vcf.certmgmt.sddcmgr.SddcManagerCertificatePlugin$$SpringCGLIB$$0.replaceCertificate(<generated>)
        at com.vmware.vcf.certmgmt.service.orch.impl.CertificateOperationOrchestratorImpl.replaceCertificate(CertificateOperationOrchestratorImpl.java:1371)
        at com.vmware.vcf.certmgmt.service.orch.impl.CertificateOperationOrchestratorImpl.lambda$createCertificateOperationTask$32(CertificateOperationOrchestratorImpl.java:541)
        at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

• Upon reviewing the /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log, the following entries were observed - 

YYYY-MM-DDTHH:MM:SS.Z ERROR [common,687fad02abcc47f9b679e918ffe47198,8c71] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-319] [1BH###] CERT_REPLACEMENT_FAILED Cannot replace existing certificate with the input cert. Validations did not pass.
Make sure the input cert chain is valid. The structure must be:
server cert followed by intermediate certs followed by CA cert
OR
A self signed server cert
All certs in the chain must conform to X.509 standards.
Also make sure that the DNS name in both the CN field and the optional Subject Alternative Name extension, is a resolvable hostname
com.vmware.evo.sddc.appliance.utilities.error.ApplianceManagerException: Cannot replace existing certificate with the input cert. Validations did not pass.
Make sure the input cert chain is valid. The structure must be:
server cert followed by intermediate certs followed by CA cert
OR
A self signed server cert
All certs in the chain must conform to X.509 standards.
Also make sure that the DNS name in both the CN field and the optional Subject Alternative Name extension, is a resolvable hostname
        at com.vmware.evo.sddc.appliance.utilities.api.rest.CertificateController.installCert(CertificateController.java:167)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
        at 

...
...
...

org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:230)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.vmware.evo.sddc.appliance.utilities.error.CertValidatorException: Error while validating certificate
        at com.vmware.evo.sddc.appliance.utilities.utils.SslCertValidator.validateCertChain(SslCertValidator.java:265)
        at com.vmware.evo.sddc.appliance.utilities.utils.SslCertValidator.validateCertsInChain(SslCertValidator.java:142)
        at com.vmware.evo.sddc.appliance.utilities.utils.SslCertValidator.performERICertValidations(SslCertValidator.java:133)
        at com.vmware.evo.sddc.appliance.utilities.NginxCertUtilityImpl.validateCert(NginxCertUtilityImpl.java:217)
        at com.vmware.evo.sddc.appliance.utilities.NginxCertUtilityImpl.installCert(NginxCertUtilityImpl.java:173)
        at com.vmware.evo.sddc.appliance.utilities.api.rest.CertificateController.installCert(CertificateController.java:165)
        ... 139 common frames omitted
Caused by: java.security.cert.CertificateException: Hostname in CN field [SDDC_FQDN] could not be resolved to an IP address of the SDDC manager [#.#.#.#]
        at com.vmware.evo.sddc.appliance.utilities.utils.SslCertValidator.validateCNAndSANDnsName(SslCertValidator.java:295)
        at com.vmware.evo.sddc.appliance.utilities.utils.SslCertValidator.validateCertChain(SslCertValidator.java:262)
        ... 144 common frames omitted

Environment

• SDDC Manager 5.2.1.0
• SDDC Manager 5.2.1.2

Cause

• This validation is failing because a recent security update upgraded the DNS Java library from version 2.1.9 to 3.6.3.
• This upgrade introduces significant API changes and incompatibilities, leading to DNS resolution issues.

Resolution

If the SDDC Manager certificate replacement fails with the above symptoms, here's how to fix it -

• Important Note - Before you start, take a snapshot of your SDDC Manager VM.

1. 1. SSH into the SDDC Manager using root user credentials.
   2. Back up the hosts file: Run the command - cp /etc/hosts /etc/hosts.bak.
   3. Edit the /etc/hosts file. Find and comment out the lines that map the SDDC Manager's Fully Qualified Domain Name (FQDN) to 127.0.0.1 and 0:0:0:0:0:0:0:1 (Both the IPv4 and IPv6 localhost entries).
   4. Retry the certificate replacement workflow that previously failed.
   5. Once the workflow finishes successfully, revert the changes you made in step 3 by uncommenting the original entries in the /etc/hosts file. This will restore the FQDN resolution mechanism to its default state for future operations.