NSX Onboarding to SSP Fails with "UPGRADE_IN_PROGRESS" Error
search cancel

NSX Onboarding to SSP Fails with "UPGRADE_IN_PROGRESS" Error

book

Article ID: 444965

calendar_today

Updated On:

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

(1) Attempting to onboard an NSX Manager instance to SSP fails or hangs during the certificate push phase.

(2) Running an upgrade pre-check from SDDC Manager, fail with consistency errors:

"NSX Upgrade Plan Consistency"... "Check if NSX upgrade plan in the upgrade coordinator is consistent"

(3) Navigating to the NSX Manager UI under System > Upgrade results in a completely blank screen with no information loaded.

(4) Reviewing the site service pod logs in SSP reveals that certificate synchronization requests initiated by SSP are explicitly rejected with the following payload:

 
HTTP Status: BAD_REQUEST
Error Code: 2190
Error Message: "A certificate batch replace operation cannot be started at this time because conflicting operations are running: UPGRADE_IN_PROGRESS. Try again later."

 

Environment

NSX 4.X , 9.x

SSP 5.x

Cause

The issue occurs during the certificate replacement workflow (/api/v1/trust-management/certificates/action/batch-replace).

Even if no manual upgrade is active, the NSX Upgrade Framework / Upgrade Coordinator service can enter a stale or hung state where it falsely reports an active UPGRADE_IN_PROGRESS state. Because certificate batch replacements are blocked during upgrades to prevent cluster state conflicts, the persistent stale state blocks SSP from completing its onboarding workflow.

Resolution

1. Verify Upgrade Status via API

To confirm if the Upgrade Coordinator is reporting a stale operation, execute the following API call via CLI:

curl -k -u "admin" -X GET "https://<nsx-manager-ip>/api/v1/upgrade/status-summary" -H "Accept: application/json"

If the service is unresponsive or explicitly stuck returning an upgrade block, proceed to the Workaround.


2. To clear the stuck upgrade state and restore responsiveness to the Upgrade Coordinator, a graceful rolling restart of the NSX Manager cluster nodes must be performed.

Follow this specific sequence to maintain cluster availability and ensure clean failover of the cluster Virtual IP (VIP):

  1. Identify the VIP Node: Determine which NSX Manager node currently holds the cluster Virtual IP (VIP).

  2. Reboot Non-VIP Nodes First: Gracefully reboot the two NSX Manager nodes that are not currently assigned the VIP.

    • Note: Allow each node to fully boot up, rejoin the cluster, and return all services to a green/healthy state before proceeding to the next node.

  3. Reboot the VIP Node Last: Gracefully reboot the final NSX Manager node (the one holding the VIP). This safely forces the VIP to failover to one of the already running, healthy nodes without interrupting cluster quorum.

  4. Post-Reboot Validation:

    • Log into the NSX Manager UI and navigate to System > Lifecycle > Upgrade to verify that the page now renders correctly.

    • Re-run the onboarding process from the SSP console or re-try the SDDC Manager upgrade pre-checks.

 

Powered by Wolken Software