Host commission fails during validation with an error "Host SSL certificate is not trusted by SDDC Manager"

Products

VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

Host commission fails for ESX host with an error message as shown below;

Host SSL certificate is not trusted by SDDC Manager

Operations manager logs reports the following log entries:

In the /var/log/vmware/vcf/operationsmanager/operationsmanager.log on SDDC Manager, you may find entries similar to:

YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] established connection with <vcenter.example.com:443.sdk>
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.e.s.c.c.v.vsphere.VsphereClient,] Successfully logged in to <https://vcenter.example.com:443.sdk>
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,] Get advanced options from VC : <vcenter fqdn>
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.s.v.p.VcCertificateModeService,] vCenter <vcenter fqdn> is in custom certificate validation mode.
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] disconnected from <vcenter.example.com:443.sdk>
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.h.c.s.i.CommissionHostsTrustValidator,] SSL certificate of host hostname.example.com will be checked to be trusted by SDDC Manager
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] opening connection to hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] established connection with hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.s.t.c.CertificateRetrieverUtil,] Certificate chain length is :1 for resource hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] disconnected from hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.c.CertificateRetrieverService,] Certificate chain validity check against current PKIXParameters failed
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.s.v.p.Tr####tHostValidatorImpl,] Hosts 'hostname.example.com' SSL Certificates are not trusted by SDDC Manager
YYYY-MM-DDTHH:MIN:SEC ERROR [vcf_om,####-###-####-#####,####] [c.v.v.h.c.s.i.CommissionHostsValidator,] Host validation failed for Host: hostname.example.com
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.c.v.vsphere.VsphereClient,] Destroying 2 open views
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] disconnected from hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] disconnected from hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,68f2a1bada43496a54807d142e77abb9,ac5a] [c.v.v.h.c.s.i.CommissionHostsValidator,om-exec-6] hostname.example.com: SDDC_TRUST_HOST_SSL_CERT_FAILED
YYYY-MM-DDTHH:MIN:SEC.322+0000 DEBUG [vcf_om,68f2a1bada43496a54807d142e77abb9,ac5a] [c.v.v.h.c.s.i.CommissionHostsValidator,om-exec-6] Completed validating Host(s).
YYYY-MM-DDTHH:MIN:SEC.322+0000 DEBUG [vcf_om,68f2a1bada43496a54807d142e77abb9,ac5a] [c.v.v.h.c.c.v.HostManagerCommDecommIsController,om-exec-6] Host validation response {"status":"error","message":"Host Validation Failed.","errors":[],"commissionHostValidationResponses":[{"ipAddress":"10.0.0.1","hostfqdn":"
hostname.example.com","hostValidationStatus":"SDDC_TRUST_HOST_SSL_CERT_FAILED","networkPoolId":"*****","networkPoolName":"ftb-ft-np01","username":"root","password":"*****","storageTypes":["VSAN_ESA"]}]}

Environment

VCF 5.2.x
VCF 9.0.0.0
VCF 9.0.1.0

Cause

SDDC Manager is unable to validate the certificate of the ESXi host during host commissioning.

This issue occurs in any of the following scenarios when commissioning a ESXi host with a default certificate:

Management domain is deployed using ESXi hosts with External certificates
Management domain is initially deployed using ESXi hosts with default certificates and later ESXi hosts certificates are replaced with External certificates.

Resolution

To commission the ESXi host, follow the below procedure depending on the how the environment is setup.

Management domain is deployed using ESXi hosts with External certificates

Note: When external certificates are used for ESXi hosts in the management domain during bring-up, all future hosts added to VMware Cloud Foundation must also use external certificates.

Replace the ESXi host default certificate with external certificate by referring to Configure ESX Hosts with Signed Certificates
Initiate host commission

Management domain is initially deployed using ESXi hosts with default certificates and later ESXi hosts certificates are replaced with External certificates.

Change the ESXi certificate mode in vCenter to vmca by referring to Change the ESX Certificate Mode
Initiate host commission
Revert the change