Certificate replacement fails with error 2190 "TRANSPORT_NODE_ONBOARDING_IN_PROGRESS" in the NSX UI

Products

• VMware NSX

Issue/Introduction

• NSX Manager certificate replacement fails through UI or Batch API replacement.

• Specific certificates, such as APH-AR or APH-TN, are expired or near expiration.
• ESXi transport nodes may show a status of HOST_DISCONNECTED or INSTALL_FAILED.

• The following error is observed when attempting to replace the certificate:

A certificate batch replace operation cannot be started at this time because conflicting operations are running: TRANSPORT_NODE_ONBOARDING_IN_PROGRESS. Try again later. (Error code: 2190)

Error screenshot as seen from NSX UI > System > Certificates > Select the certificates you want to replace > Actions > Replace Certificates :

Environment

VMware NSX
VMware VCF NSX 

Cause

During certificate replacement a check is made against transport node states, this check is incorrectly blocking certificate replacement when hosts are in some specific non healthy states.

Resolution

This issue is resolved in VMware NSX 4.2.4 and VCF NSX 9.1.0 , available at Broadcom downloads.

If you are having difficulty finding and downloading software, review the Download Broadcom products and software KB.

Workaround

In order to temporarily resolve the issue, you can also follow the below steps:

1. Identify any stale or disconnected ESXi transport nodes in the NSX inventory.
2. Rectify any host related errors
3. Reattempt the NSX Manager UI certificate replacement from GUI or API.
4. Verify success of replacement and NSX Manager accessibility.

Note: If "Replace Certificate" option from an expired certificate is giving Batch Replace error, "Apply Certificate" for the respective service from new certificate can be attempted.

Also, the certificate can be replaced without using the batch replace process by replacing the certificate for each API/VIP service. Beginning with Step #1 in the following document: Replace Certificates Through API

As an alternative, you can use the CARR script to replace the certificates.  This is recommended if the above procedures aren't working or if you have additional certificates that need to be replaced because they are expired or are expiring soon.  Please reference KB 369034 - Using Certificate Analyzer, Results, and Recovery (CARR) script to fix certificate related issues in NSX.