NSX Compute Manager 'Connection Status' Down

Products

VMware NSX VMware vCenter Server VMware NSX Data Center for vSphere VMware Cloud Foundation

Issue/Introduction

Compute Manager 'Connection Status' in NSX shows Down.

If you attempt to edit the Compute Manager and click 'Save', you may encounter an error similar to one of the following:

Computer manager <name> with Id <ID> connection config is invalid. Edit Hostname and provide computer manager credentials. (Error code: 7055)
Certificate Chain of Computer Manager <name> is invalid. Please check Issuer and subject in the chain. (Error code: 90204)

In NSX 9.0+, an error message may state:

Unable to connect to the compute manager <vCenter_name> as its trusted root certificate cannot be found. Validate the certificate chain in vCenter and then re-register the compute manager by editing it.

The last inventory update date could match these scenarios:
- Environment may have been upgraded recently (NSX-T Data Center and/or vCenter).
- vCenter Certificates may have been changed recently.

You may see messages similar to the following in the /var/log/cm-inventory/cm-inventory.log file on the NSX manager node:

<timestamps>  INFO http-nio-127.0.0.1-7443-exec-2 NsxTrustManagerBinding - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Try create  TrustManager of type PKIX
<timestamps>  WARN http-nio-127.0.0.1-7443-exec-2 VcUtilsImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="cm-inventory"] IOException occurred
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.

If the vCenter Server certificate was recently replaced with a custom CA-signed certificate, you may see messages similar to the following in the /var/log/cm-inventory/cm-inventory.log file on the NSX manager node:

<timestamps> ERROR http-nio-127.0.0.1-7443-exec-2 VcPlugin 4732 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP40219" level="ERROR" reqId="11111111-2222222-7e7e7e7e7" subcomp="cm-inventory" username="admin"] Certificate of Vc example.com is invalid. It might be caused by issuer not being same as subject of next certificate in certificate chain.

If you attempt to add a new computer manager to an existing NSX cluster, you may see messages similar to the following in the /var/log/cm-inventory/cm-inventory.log file on the NSX manager node:

<timestamps> ERROR http-nio-127.0.0.1-7443-exec-3 VcPlugin - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP40106" level="ERROR" subcomp="cm-inventory"] Unable to login with username password for <IP/FQDN>
com.vmware.vim.vmomi.client.exception.SslException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found.

Environment

VMware NSX-T Data Center
VMware NSX

VMware vCenter Server

VMware Cloud Foundation 9 (VCF 9)

Cause

The Connection Status is down due to an invalid or incorrect vCenter Server Machine SSL certificate chain.

A correct chain consists of:

Server/Leaf Certificate (vCenter Server)
Intermediate Certificate
Root Certificate

Resolution

The vCenter Machine SSL certificate chain needs to be checked and fixed.

*The CARR script can be run to see if it detects a thumbprint or chain order issue. See Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX.

Manual validation can be performed using the following steps:

Obtain the Machine SSL certificate chain from vCenter Server using below command:
```
openssl s_client -showcerts -debug -connect <VC-IP>:443
```
Validate the certificate chain using any certificate checking resource.
Chain the certificate correctly using the format Leaf/Server->Intermediate->Root
Use a file transfer utility to copy the correct certificate chain to the /tmp directory on the vCenter Server
Follow Replace VC Machine SSL certificate with Custom CA Signed Certificate
Edit the Compute Manager and click 'Save'.
Confirm the Connection Status shows 'Up'.

*Note: When a custom CA-signed certificate is in use on the vCenter Server, you must engage the CA vendor to issue a new certificate.

===== Resolution for SDDC Manager and VCF 9 environments.

In VCF 9, SDDC Manager will be used to manage certificate for vCenter Server. Follow the steps based on the below documentation to update the custom CA-signed machine SSL certificate on vCenter Server. Then, similar to the steps above, Edit the Compute Manager in the NSX Manager UI and click Save to update it.

Additional Information

Reusing a custom signed leaf certificate for the updated chain :
The vCenter UI and the /usr/lib/vmware-vmca/bin/certificate-manager script do not allow you to reapply an identical Leaf Certificate.
Using the vCert tool for the same action, if it succeeds, may also result in the same error continuing to show on NSX.

To successfully apply the same Leaf Certificate with a corrected chain, a temporary intermediary step is required.

Replace the current Machine SSL certificate with a temporary VMCA-signed certificate.
Replace the certificate again using the original Leaf Certificate and the correct, updated chain.

Replacing vSphere Certificates

After the vCenter Server certificate is replaced, the compute manager connection is "Down" in the NSX UI