Cannot upgrade NSX host component after ESXi is rolled back
search cancel

Cannot upgrade NSX host component after ESXi is rolled back

book

Article ID: 370357

calendar_today

Updated On: 01-26-2025

Products

VMware NSX

Issue/Introduction

  • NSX Upgrade had completed sucessfully.
  • ESXi is rolled back after the successful NSX upgrade, and NSX host components are pre-upgrade version.
  • When you upgrade the host component again on NSX Manger, it fails.
  • You see a notification pops up "Upgrade continue operation failed: Cannot continue upgrade. Upgrade coordinator backend is busy in sync operation. Please try again after some time."

Environment

NSX-T

Cause

Host component can not be upgraded if OverallUpgradeStatus is SUCCESS.

You can check /var/log/upgrade-coordinator/upgrade-coordinator.log in the orchestrator node and verify OverallUpgradeStatus causes the failure.

<Timestamp>  INFO http-nio-127.0.0.1-7442-exec-[N] UpgradeCoordinatorFacadeImpl 2975 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Invoke Continue upgrade with Parameters com.vmware.nsxapi.upgrade.dto.ContinueRequestParameters@57c4a0a5[skip=false,hasSkip=true,componentType=HOST,hasComponentType=true]
...
<Timestamp>  INFO http-nio-127.0.0.1-7442-exec-[N] UpgradeServiceImpl 2975 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Got OverallUpgradeStatus as SUCCESS from DB
<Timestamp>  INFO http-nio-127.0.0.1-7442-exec-[N] FacadeInterceptorHelperImpl 2975 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Caught error in facade interceptor
com.vmware.nsx.management.upgrade.exceptions.UpgradeNotPausedException: null

Resolution

You can reinitialize OverallUpgradeStatus by restarting upgrade coordinator.

  1. Identify the orchestrator node.

    Run get service install-upgrade on a NSX Manager.

    nsx01> get service install-upgrade
    Thu Jun 20 2024 UTC 02:23:20.376
    Service name:      install-upgrade
    Service state:     running
    Enabled on:        <The orchestrator node>

    You can identify the orchestrator node as Enabled on:.

  2. Restart upgrade-coordinator on the orchestrator node.

    Run restart service install-upgrade on the orchestrator node.

  3. Now you can upgrade the host.