Unable to issue VMCA signed certificate for NSX manager - workload domain deployment fails in VCF 9.0

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

Workload domain creation fails after deploying NSX Manager appliances. The subtask fails while issuing the VMCA certificate for the NSX Managers.
The following error is observed in the Task Creating Workload Domain:

^Description

^{Issue VMCA Certificate for NSX Managers}

^{Progress Messages}

^{Unable to issue VMCA signed certificate for NSX Manager NSX_FQDN}

^Error

^{Message: Unable to issue VMCA signed certificate for NSX Manager NSX_FQDN}

^{Remediation Message:}
^{Reference Token:}

^{Cause: Retriable operation 'Issuing certificate to NSX manager : 'NSX_FQDN' failed to complete after 3 retries.}
^{InvalidRequest(com.vmware.vapi.std.errors.invalid_request) (statusCode:400) => { messages= [], data => {error_message=Certificate chain validation failed. Make sure avalid chain is provided in order leaf,intermediate,root certificate.,}
^{httpStatus=BAD_REQUEST, error_code=2076, module_name=internal-framework}, errorType = INVALID_REQUEST }}

/var/log/vmware/vcf/domainmanager.log:

^{yyyy-mm-ddThh:mm:ss DEBUG [vcf_dm] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-9] Created ApiClient connection to: nsxt_fqdn}
^{yyyy-mm-ddThh:mm:ss DEBUG [vcf_dm] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-9] NSX Version: 9.0.2.0.25150386, NSX Version with policy baseline: 4.1.0.0.0-0}
^{yyyy-mm-ddThh:mm:ss DEBUG [vcf_dm] [c.v.v.c.n.s.c.c.NsxtManagerCertOperations,dm-exec-9] importing certificate with the id: 707e5aab-b34a-4d0f-beba-d737584cbe4c}
^{yyyy-mm-ddThh:mm:ss ERROR [vcf_dm] [c.v.v.c.n.s.c.c.ComplexHelpers,dm-exec-9] Exception occurred during NSX API invocation}
^{java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.InvalidRequest: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) (statusCode:400) => {}
^{messages = [],}
^{data = => {error_message = Certificate chain validation failed. Make sure a valid chain is provided in order leaf,intermediate,root certificate., httpStatus=BAD_REQUEST, error_code=2076, module_name=internal-framework},}
^{errorType = INVALID_REQUEST}
^}
^...
^...
^...
^{yyyy-mm-ddThh:mm:ss DEBUG [vcf_dm] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-9] Closed ApiClient connection.}
^{yyyy-mm-ddThh:mm:ss ERROR [vcf_dm] [c.v.v.c.f.p.n.a.IssueVMCACertsForNsxtManagerAction,dm-exec-9] UNABLE_TO_ISSUE_VMCA_SIGNED_CERTIFICATE}^{java.lang.RuntimeException: Retriable operation 'Issuing certificate to NSX manager: nsxt_fqdn' failed to complete after 3 retries.}
NSX requires a full chain consisting of the Leaf (vCenter), Intermediate(s), and Root certificates in that specific order. "Certificate chain validation failed" typically indicates that the vCenter Server's Machine SSL certificate chain is incomplete or incorrectly ordered. Validation performed using Certificate chain validation failed with error 2076 due to incorrect certificate order confirms the existing chain topology is correct.

Environment

VCF 9.x

Cause

Management vCenter Server is configured with the VMCA acting as an intermediate Certificate Authority. During a new Workload Domain deployment, SDDC Manager delegates certificate generation to the Management Domain vCenter Server's VMCA. This VMCA signs the certificates for the new NSX Manager cluster. If the management VCSA is configured with VMCA as an intermediate CA, this results in an unsupported configuration for this workflow.

VMware Cloud Foundation (VCF) 9.x does not support configuring the VMCA as an intermediate CA - VMware Cloud Foundation (VCF) supports using the VMCA as an intermediate CA.

Resolution

1. Reset the Management vCenter Server VMCA certificates to default.

Take powered-off snapshots of all the vCenters in the VCF infra. If the infrastructure has a single vCenter or isolated VI domains, then take a snapshot without memory in the powered-on state.
To reset the certificate, follow - Reset All Certificates Using the Certificate Manager.

2. Re-trust the Management vCenter Server certificate within SDDC Manager.

Once the vCenter is reset to defaults from the previous steps, the new root CA of the vCenter needs to be added to the SDDC manager's trust store
Follow the steps from the KB article: How to import the vCenter root certificate into the SDDC manager TrustStore

3. Retry the failed Workload Domain deployment workflow.

On the GUI, navigate to the Tasks pane.
Locate the failed task for Creating Workload Domain and click on Restart Task.

Note: To replace the vCenter with Custom certificates - Managing Certificates in VMware Cloud Foundation