NSX Manager Upgrade Fails Due to Incorrect Certificate Chain on vCenter Port 8089

Products

• VMware NSX
• VMware vCenter Server

Issue/Introduction

This article provides a resolution for NSX Manager upgrade failures occurring during VCF upgrades, specifically when the failure is attributed to SSL/TLS authentication errors on vCenter Server port 8089.

When attempting an NSX Manager upgrade via SDDC Manager, the upgrade process fails during post-checks. The /var/log/cm-inventory/cm-inventory.log on the NSX Manager shows a TLS handshake failure during the loginByCertificateAsExtensionOneCall method with errors similar to:

• Error occurred during login by certificate as extension for cm
• com.vmware.vim.vmomi.client.exception.SslException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
• java.security.cert.CertificateException: Unable to construct a valid chain
• java.security.cert.CertPathBuilderException: Unable to find certificate chain.

vCenter vpxd log also be flooded with log lines 

• vpxd [vc@4413 sub="IO.Connection"] Failed to SSL handshake; SSL(<io_obj t:N7Vmacore6System19TCPSocketObjectAsioE ###(sslv3 alert certificate unknown (SSL routines)), duration: 10msec

These symptoms are consistent with those documented in KB 454127.

Environment

VMware vCenter 9.1.x

VMware NSX 9.1.x

Cause

The vpxd service contains a defect where it fails to correctly parse the full multi-block PEM certificate chain when serving traffic on port 8089. Instead of using the correct, complete chain, vpxd only reads the leaf certificate and subsequently falls back to the local OS certificate store (/etc/ssl/certs) to complete the chain.

If an incorrect, stale, or conflicting root certificate exists within /etc/ssl/certs that shares common attributes (such as the same Public Key or Common Name) with the expected intermediate CA, vpxd may erroneously complete the chain using this incorrect certificate. This resulting in invalid certificate chain being presented on port 8089, causing SSL/TLS authentication failures with NSX Manager.

Resolution

The recommended workaround is to remove the conflicting or stale certificate from the local OS certificate store, forcing vpxd to handle the chain properly.

Workaround Procedure

1. Log in to the affected vCenter Server via SSH.
2. Identify the conflicting certificate in /etc/ssl/certs based on the diagnostic output.
3. Back up the certificate file to a secure location: cp /etc/ssl/certs/<filename>.pem /root/<filename>.pem.bak
4. Delete the conflicting certificate file: rm -f /etc/ssl/certs/<filename>.pem
5. Restart the vpxd service to apply the changes: service-control --restart vpxd

Additional Information

To confirm this issue, compare the certificate chain served on port 8089 with port 443 to identify discrepancies. Run the following commands from the vCenter Server CLI:

openssl s_client -connect localhost:8089 -showcerts </dev/null 2>/dev/null > port8089.crt

openssl s_client -connect localhost:443 -showcerts </dev/null 2>/dev/null > port443.crt

Examine port8089.crt and port443.crt to determine if port 8089 is presenting an incorrect or incomplete certificate chain.