LCM Upgrade error: NSX manager upgrade failed in waitForNsxtComponentUpgradeToComplete. Overall upgrade status is :PAUSING. componentUpgradeStatus for NSX Manager is: FAILED

Products

• VMware NSX

Issue/Introduction

• Upgrading NSX (e.g., from 9.1.0.0100 to 9.1.0.0200) pauses or fails at the MP_NOTIFY_COMPLETION stage.

• The NSX Upgrade UI displays the following error: Unexpected error while upgrading upgrade unit: Upgrade failed at mp_notify_completion_uu_name with error : LM component upgrade has failed with error code: 21, please perform the following remediation: Service account is not enabled in one or more compute managers.

• The /var/log/cm-inventory/cm-inventory.log on the NSX Manager shows a TLS handshake failure during loginByCertificateAsExtensionOneCall with errors similar to:

   

  • com.vmware.vim.vmomi.client.exception.SslException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)

  • java.security.cert.CertificateException: Unable to construct a valid chain

  • java.security.cert.CertPathBuilderException: Unable to find certificate chain.

Environment

• NSX 9.1.0.100

Cause

vCenter Server is presenting inconsistent Machine SSL certificates on its vAPI/REST endpoint (port 443) and its VMOMI/SDK endpoint (port 8089).

This mismatch typically occurs when a custom 3rd-party Machine SSL certificate is applied to vCenter, but the certificate replacement process fails to notify the applmgmt service (often due to an HTTP 403 Forbidden authorization issue). Because applmgmt fails, vpxd is never notified to reload the new certificate. As a result, port 443 serves the new trusted certificate (allowing the NSX Service Account login to succeed), while port 8089 continues to serve the old, stale certificate (causing the NSX extension certificate login to fail).

Before running the resolve steps, check if there is a mismatch between port 443 and 8089 certificates on the vCenter:

1. SSH into the vCenter VM. 
2. Run the following:
   1. openssl s_client -connect localhost:8089 -showcerts </dev/null 2>/dev/null > port8089.crt
   2. openssl s_client -connect localhost:443-showcerts </dev/null 2>/dev/null >  port443.crt
3. The contents of both files must be identical
   1. If not identical, follow the resolution steps. 

Resolution

Here are the steps to resolve this issue and force vCenter to load the new Machine SSL certificate across all services (including the sdktunnel on port 8089):

1. Restart all vCenter Server services.

2. Verify that the correct certificate is now being presented on port 8089 using the following command on the vCenter Server: openssl s_client -connect localhost:8089 -showcerts </dev/null

3. Resume or retry the NSX upgrade from the UI.