Host commission fails during validation with an error "Host SSL certificate is not trusted by SDDC Manager"

Products

VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

Host commission fails for ESX host with an error message as shown below;

Host SSL certificate is not trusted by SDDC Manager

Operations manager logs reports the following log entries:

/var/log/vmware/vcf/operationsmanager/operationsmanager.log

YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.VCFSupportedESXiVersionValidator,] ESXi version on host hostname.example.com is 9.0.0.0.24755229
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.VCFSupportedESXiVersionValidator,] ESXi version of the Host - [hostname.example.com] is : 9.0.0.0.24755229
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.u.HostFreePoolValidationUtils,] Get VCF supported ESXi versions from LCM productType: ESX and scope: DEPLOYMENT
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.VCFSupportedESXiVersionValidator,] VCF supported ESXi versions: [9.0.0.0.24755229]
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.VCFSupportedESXiVersionValidator,] Host version: 9.0.0.0.24755229
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.VCFSupportedESXiVersionValidator,] ESXi version: 9.0.0.0.24755229 is VCF supported. Validation successful for Host: hostname.example.com
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.HostVsanPartitionValidator,] Host hostname.example.com storage type [VSAN_ESA] is not VSAN, the VSAN based validations are not applicable
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.util.ValidationUtilImpl,] The ESXi version for the host hostname.example.com: 9.0.0
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.InstalledVibsValidator,] Host hostname.example.com doesn't have undesired vibs on it, validation success.
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.DeployedVMsValidator,] Validating any VMs exists on host hostname.example.com
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.e.s.c.util.ValidationUtilImpl,] Validating if there are any VMs in host hostname.example.com
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.e.s.c.util.ValidationUtilImpl,] Successfully validated that there are no VMs on host hostname.example.com
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.FibreChannelStorageValidator,] validation started for this host hostname.example.com and storage type [VSAN_ESA]
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.FibreChannelStorageValidator,] Fiber channel based validations are not applicable for this given Host hostname.example.com and storage type [VSAN_ESA]
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.HostVsanEsaVersionValidator,] Validating if host version is supported for vSAN ESA for host hostname.example.com.
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.v.h.f.v.s.i.HostVsanEsaVersionValidator,] Successfully verified version for vSAN ESA for host hostname.example.com.
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,] Fetching Management vCenter data from inventory
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,] Fetching vCenters data from inventory
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.c.f.p.n.s.s.CredentialMgmtServiceImpl,] Feature flag feature.vcf.service.accounts.vcenter.vcf.integration is enabled
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.c.f.p.n.s.s.CredentialMgmtServiceImpl,] Using service account credentials for vCenter ID ####-###-####-#####
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.s.s.ServiceCredentialsHelper,] Getting credentials for target type VCENTER, entity ID ####-###-####-##### and service type SDDC_MANAGER
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,] Security config retrieved {"fipsMode":true}
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,] Connecting to <https://vcenter.example.com:443.sdk>
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] opening connection to <vcenter.example.com:443.sdk>
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.s.t.DynamicTr####tManager,] Checking validity of certificate chain CN=<vcenter fqdn>, OU=##, O=####, L=####, ST=####, C=####, SerialNumber=####-###-####-#####,CN=####, DC=####, DC=####, SerialNumber=####-###-####-#####
YYYY-MM-DDTHH:MIN:SEC.269+0000 DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.s.t.DynamicTr####tManager,] Certificate chain CN=<vcenter fqdn>, OU=##, O=####, L=####, ST=####, C=####, SerialNumber=####-###-####-#####,CN=####, DC=####, DC=####, SerialNumber=####-###-####-##### is valid
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] established connection with <vcenter.example.com:443.sdk>
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.e.s.c.c.v.vsphere.VsphereClient,] Successfully logged in to <https://vcenter.example.com:443.sdk>
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,] Get advanced options from VC : <vcenter fqdn>
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.s.v.p.VcCertificateModeService,] vCenter <vcenter fqdn> is in custom certificate validation mode.
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] disconnected from <vcenter.example.com:443.sdk>
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.v.h.c.s.i.CommissionHostsTrustValidator,] SSL certificate of host hostname.example.com will be checked to be trusted by SDDC Manager
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] opening connection to hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] established connection with hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.s.t.c.CertificateRetrieverUtil,] Certificate chain length is :1 for resource hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] disconnected from hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.c.CertificateRetrieverService,] Certificate chain validity check against current PKIXParameters failed
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.s.v.p.Tr####tHostValidatorImpl,] Hosts 'hostname.example.com' SSL Certificates are not trusted by SDDC Manager
YYYY-MM-DDTHH:MIN:SEC ERROR [vcf_om,####-###-####-#####,####] [c.v.v.h.c.s.i.CommissionHostsValidator,] Host validation failed for Host: hostname.example.com
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,####-###-####-#####,####] [c.v.e.s.c.c.v.vsphere.VsphereClient,] Destroying 2 open views
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] disconnected from hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC INFO [vcf_om,####-###-####-#####,####] [o.b.jsse.provider.ProvTlsClient,] [client #### @####] disconnected from hostname.example.com:443
YYYY-MM-DDTHH:MIN:SEC DEBUG [vcf_om,68f2a1bada43496a54807d142e77abb9,ac5a] [c.v.v.h.c.s.i.CommissionHostsValidator,om-exec-6] hostname.example.com: SDDC_TRUST_HOST_SSL_CERT_FAILED
YYYY-MM-DDTHH:MIN:SEC.322+0000 DEBUG [vcf_om,68f2a1bada43496a54807d142e77abb9,ac5a] [c.v.v.h.c.s.i.CommissionHostsValidator,om-exec-6] Completed validating Host(s).
YYYY-MM-DDTHH:MIN:SEC.322+0000 DEBUG [vcf_om,68f2a1bada43496a54807d142e77abb9,ac5a] [c.v.v.h.c.c.v.HostManagerCommDecommIsController,om-exec-6] Host validation response {"status":"error","message":"Host Validation Failed.","errors":[],"commissionHostValidationResponses":[{"ipAddress":"10.0.0.1","hostfqdn":"
hostname.example.com","hostValidationStatus":"SDDC_TRUST_HOST_SSL_CERT_FAILED","networkPoolId":"*****","networkPoolName":"ftb-ft-np01","username":"root","password":"*****","storageTypes":["VSAN_ESA"]}]}

Environment

VCF 9.0.0.0

VCF 9.0.1.0

Cause

When you install ESX software on a server to create an ESX host, the host initially has an autogenerated certificate. By default, when the ESX host is added to a vSphere cluster during deployment of the management domain or other operations involving ESX hosts (for example, host commissioning, workload domain creation, and so on), the autogenerated certificate is replaced with a certificate that is signed by the VMware Certificate Authority (VMCA). When you use external certificates during deployment, they are not replaced by VMCA-signed certificates. Once you perform deployment with external certificates for ESX hosts, all future ESX hosts added to VCF must also use external certificates.

Resolution

Its a 3 step process to commission the host with Custom CA certificate.

Replace default ESX certificate by follow the steps mentioned in Replacing the Default ESX Certificate with a Custom Certificate or Adding Custom Certificate on ESXi hosts through CLI
Once ESXi has been replaced with Custom CA certificate and successfully connected to vCenter server, proceed to Add Custom CA certificate to SDDC Manager trust store
Initiate Host commission.