Error: Configure NSX Fabric Compute Manager fails due to incomplete certificate chain in VCF 9.x

Products

VMware vCenter Server VMware Cloud Foundation VMware NSX

Issue/Introduction

During a VMware Cloud Foundation (VCF) 9.x deployment, the "Configure NSX Fabric Compute Manager" task fails. The deployment UI displays an error similar to:

Configure NSX Fabric Compute Manager Failed
Unable to configure fabric for vcenter.example.com on nsx.example.com

/var/log/vmware/vcf/domainmanager/domainmanager.log (SDDC Manager)

`Exception occurred during NSX API invocation java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.InvalidRequest: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) (statusCode:400) => { messages = [], data = => {error_message=Certificate chain of Compute Manager vcenter.example.com is invalid. Please check Issuer and Subject in the chain., httpStatus=BAD_REQUEST, error_code=90204, module_name=inventory-mgmt}, errorType = INVALID_REQUEST}`

/var/log/cm-inventory/cm-inventory.log (NSX):

`Certificate error while connecting to Compute Manager com.vmware.vim.vmomi.client.exception.SslException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found.`

When attempting a vCenter Server Import to VCF, the import gets stuck because the VMware NSX Managers cannot add a compute manager. The UI displays the following error:
Description: Configure NSX Fabric Compute Manager
Message: Unable to configure fabric for #### on ####.
Remediation Message: Check if vCenter is up and running also check if NSX Extension present in vCenter Server and delete the same. Retry the failed workflow.
Reference Token: ####
Cause: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) (statusCode:400) => { messages = [], data = => {error_message=Certificate chain of Compute Manager #### is invalid. Check Issuer and Subject in the chain., httpStatus=BAD_REQUEST, error_code=90204, module_name=inventory-mgmt}, errorType = INVALID_REQUEST }
Similar issue may occur during Convert operation using VCF 9.x Installer and "Deploy and Configure NSX" task fails with below Error ( Failed Sub-Task in execution will reflect as : "Generate input for trusting certificates" ):

Unable to configure fabric for <vCenter Server IP/FQDN> on <NSX-VIP-IP/FQDN>. Remediation: Please check if vCenter is up and running also check if NSX Extension present in vCenter Server and delete the same. Retry the failed workflow. Reference Token: N####0

Environment

VMware Cloud Foundation (VCF) 9.x
vCenter Server
NSX-T

Cause

The root cause of this failure is that NSX is unable to trust the vCenter machine_ssl certificate chain. This typically occurs because the vCenter's `/etc/vmware-vpx/ssl/rui.crt` file contains only the root and leaf certificates, but is missing one or more intermediate certificates in the chain.

Resolution

Ensure there is a proper snapshot or backup of the vCenter before proceeding.

To resolve this issue, the vCenter machine_ssl certificate chain must be recreated to include the full chain (leaf, intermediate(s), and root certificates) and then replaced on the vCenter Server.

Scenario 1

If the thumbprint of the vCenter certificate will be changed, follow the steps in Replace vCenter Machine SSL certificate Custom Certificate Authority Signed Certificate to replace the certificate.

Important Note: If Scenario 1 resolution is applied while not initiating the installer wizard from scratch, a mismatch will occur between the new vCenter SSL thumbprint and the old SSL thumbprint already present in the workflow.json file.
Apply the resolution provided in 'Configure NSX Fabric Compute Manager Task' fails in VCF 9.0 deployment due to a vCenter SSL THUMBPRINTS MISMATCH.

Note: VMware Cloud Foundation (VCF) 9.x does not support configuring the vCenter Server VMCA as an intermediate Certificate Authority. KB for reference: VMCA as an intermediate CA

Scenario 2

If the thumbprint of the vCenter certificate will NOT be changed, use the following steps.

Method A: Using vCert Script

Download the vCert script: vCert - Scripted vCenter certificate replacement.
Run the script and select Option 3 (Manage certificates).
Select Option 1 (Machine SSL certificate).
Provide the path to the complete .pem file formatted as: Leaf > Intermediate(s) > Root.

Method B: Manual Replace via vecs-cli

Obtain the full certificate chain: Ensure the complete certificate chain (including all intermediate certificates and the root certificate) for the vCenter Server's machine_ssl certificate is concatenated into a single `.crt` file.

The machine_name_ssl.cer should be a complete chain file similar to the order below:

SSH into the vCenter Server Appliance.
Backup Existing Certificate and Key:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT > ~/backup_machine.cer
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT > ~/backup_machine.key
Delete Existing Certificate:
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CERT -y
Import New Certificate with Full Chain:
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert <path_to_full_chain_cert.cer> --key ~/backup_machine.key Note: Replace `<path_to_full_chain_cert.cer>` with the actual path to your certificate file (e.g., `/tmp/full_chain_cert.cer`).
Restart vCenter Services:
service-control --stop --all && service-control --start --all
Retry Failed VCF Deployment Workflow:
Note: If unable to get back to the VCF Installer UI, reference VCF Installer UI converts to SDDC Manager UI before deployment is complete

Additional Information

There are other possible causes for this task to fail. Troubleshooting steps can be found in the following KB:

NSX error "Certificate chain of compute manager "VC-FQDN"/"VC-shortname" is invalid (Error code: 90204)"

if further assistance is required, see Contact Broadcom support.