installing Signed Certificates on SDDC Manager fails with error 'Validation failed. Throwing error from session middleware -getaddrinfo ENOTFOUND localhost'

Products

• VMware SDDC Manager

Issue/Introduction

• SDDC Manager certificate installation failed during validation -

• Certificate chain is valid and the DNS name in both the CN field and the optional Subject Alternative Name extension, is a resolvable hostname. 

• Observation in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log - 

  YYYY-MM-DDTHH:MM:SS.Z ERROR [vcf_om,<id>] [c.v.v.c.s.SddcManagerCertificatePluginService,om-exec-17] SDDC Manager Certificate Replacement failed:
  com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException:
  YYYY-MM-DDTHH:MM:SS.Z ERROR [vcf_om,<id>][c.v.v.c.s.SddcManagerCertificatePlugin,om-exec-22] SDDC Manager Certificate Replacement failed: java.net.ConnectException: Failed to connect to localhost/[#:#:#:#:#:#:#:1]:7100  <--IPv6
  com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException: java.net.ConnectException: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:7100

• In /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log, the following entries were observed - 

  YYYY-MM-DDTHH:MM:SS.Z ERROR [common,687fad02abcc47f9b679e918ffe47198,8c71] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-319] [1BH###] CERT_REPLACEMENT_FAILED Cannot replace existing certificate with the input cert. Validations did not pass.
  Make sure the input cert chain is valid. The structure must be:
  server cert followed by intermediate certs followed by CA cert
  OR
  A self signed server cert
  All certs in the chain must conform to X.509 standards.
  Also make sure that the DNS name in both the CN field and the optional Subject Alternative Name extension, is a resolvable hostname
  com.vmware.evo.sddc.appliance.utilities.error.ApplianceManagerException: Cannot replace existing certificate with the input cert. Validations did not pass.
  Make sure the input cert chain is valid. The structure must be:
  server cert followed by intermediate certs followed by CA cert
  OR
  A self signed server cert
  All certs in the chain must conform to X.509 standards.
  Also make sure that the DNS name in both the CN field and the optional Subject Alternative Name extension, is a resolvable hostname

Environment

• VMware SDDC Manager 5.2.x

Cause

• This error occurs because the commonsvcs API, called during the replacement process, validates the FQDN resolution of the input certificate.
• This validation is failing because a recent security update upgraded the DNS Java library from version 2.1.9 to 3.6.3.
• This upgrade introduces significant API changes and incompatibilities, leading to DNS resolution issues.

Resolution

• Note - Before you start, take a snapshot of your SDDC Manager VM.
  • SSH into the SDDC Manager using root user credentials.
  • Back up the hosts file: Run the command - cp /etc/hosts /etc/hosts.bak.
  • Edit the /etc/hosts file. Find and comment out the lines that map the SDDC Manager's Fully Qualified Domain Name (FQDN) to 127.0.0.1 and ::1 (Both the IPv4 and IPv6 localhost entries).
  • Add the /etc/hosts file with the entries using the shortname instead of FQDN:
    127.0.0.1 SDDC_Manager_Short_Name localhost
    ::1 SDDC_Manager_Short_Name localhost ipv6-localhost ipv6-loopback
  • Retry the certificate replacement workflow that previously failed.
  • Once the workflow finishes successfully, revert the changes you made in step 3.