Unable to add ESXi hosts with custom CA signed certificates into a cluster from SDDC Manager.
search cancel

Unable to add ESXi hosts with custom CA signed certificates into a cluster from SDDC Manager.

book

Article ID: 327205

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

Unable to add ESXi hosts with custom CA signed certificates into a cluster from SDDC Manager. 

Similar error is encountered in the SDDC UI. 

Failed to add host <hostname> to Datacenter <dcname>
Remediation Message:
Reference Token: 2GSDH3
Cause:
Type: com.vmware.evo.sddc.common.core.error.LocalizableRuntimeException
Message: Failed running operation with vSphere @ <vcenterfqdn>
Type: java.lang.IllegalArgumentException
Message: Illegal base64 character 2d

Similar error is encountered in the domainmanager.log

/var/log/vmware/vcf/domainmanager/domianmanager.log
2022-08-26T15:48:48.434+0000 DEBUG [vcf_dm,100ffa4331d341d1,4c97] [c.v.e.s.s.t.c.CertificateRetrieverUtil,dm-exec-16]  Certificate chain length is :1 for resource <hostname>:443
2022-08-26T15:48:48.440+0000 DEBUG [vcf_dm,100ffa4331d341d1,4c97] [c.v.e.s.c.c.CertificateRetrieverService,dm-exec-16]  Certificate chain validity check against current PKIXParameters failed
java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:153)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
        at com.vmware.evo.sddc.common.certificateutil.CertificateRetrieverService.getTrustedCertificateValidatingChain(CertificateRetrieverService.java:83)
        at com.vmware.evo.sddc.common.certificateutil.CertificateRetrieverService.getTrustedCertificateFromServer(CertificateRetrieverService.java:52)
        at com.vmware.evo.sddc.common.services.validation.pki.TrustHostValidatorImpl.lambda$checkVcTrustHostCertificate$2(TrustHostValidatorImpl.java:80)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)



Environment

VMware Cloud Foundation 3.x
VMware Cloud Foundation 4.x

Cause

A known limitation where SDDC leverages the vmca cert.mode to add ESXi hosts to vSphere clusters.

Resolution

1.Generate a VMCA signed certificate from the MGMT vCenter for the ESXi host. 

  • SSH into the MGMT vCenter.
  • Navigate to the tmp directory
  • Copy or scp the vmca_host_cert script to the vCenter.
  • chmod the script
chmod +x vmca_host_cert.sh
  • Run the vmca_host_cert script.
./vmca_host_cert.sh <theesxihostname>
Status : Success
Using config file : /usr/lib/vmware-vmca/share/config/certool.cfg
Status : Success
2.Copy the generated certificate and key from vCenter to the /etc/vmware/ssl directory on the ESXi host. 
-rw-------  1 vmcad-user lwis 1.7K Jul 13 15:52 <theesxihostname>.key
-rw-r--r--  1 vmcad-user lwis 1.6K Jul 13 15:52 <theesxihostname>.crt     

3.Apply the VMCA signed certificates to the ESXi host. 

  • SSH into the ESXi host
  • Navigate to the /etc/vmware/ssl directory.
cd /etc/vmware/ssl
  • Rename the existing certificates using the following commands
mv rui.crt orig.rui.crt
mv rui.key orig.rui.key
  • Rename the new certificate and key to rui.crt and rui.key.
mv <theesxihostname>.crt rui.crt
mv <theesxihostname>.key rui.key
  • Persist the changes into the system disk 
/sbin/auto-backup.sh
  • Cycle the ESXi services.
services.sh restart

4.Add the ESXi host into the cluster via SDDC Manager. 

5.Apply the custom CA signed certificates to the ESXi host, Ref KB:  Adding Custom Certificate on ESXi hosts through CLI

Attachments

vmca_host_cert get_app