Unable to upgrade NSX components in SDDC Manager. Error: NSX Manager is in error state due to audit failure. Please run upgrade pre-checks before proceeding with upgrade

Products

• VMware SDDC Manager

Issue/Introduction

The following error is seen when upgrading NSX components in a Workload Domain through SDDC Manager.

NSX Manager is in error state due to audit failure. Please run upgrade pre-checks before proceeding with upgrade

The below conditions apply:

• In NSX Manager, the Pre-Upgrade Bundle(PUB) file was uploaded and a Pre-check was run.
• The output of the following command on the SDDC Manager does not list any component versions, curl -k -u admin -X GET -H "Content-Type:application/json" https://<NSX MANAGER VIP FQDN>/api/v1/upgrade/summary
  
  

  curl -k -u admin -X GET -H "Content-Type:application/json" https://<NSX MANAGER VIP FQDN>api/v1/upgrade/summary
  Enter host password for user 'admin':
  {
    "system_version" : "4.1.2.5.0.24150840",
    "upgrade_coordinator_version" : "4.1.2.6.0.24723857",
    "pre_upgrade_bundle_version" : "4.1.2.6.0.24723849",
    "upgrade_status" : "NOT_STARTED",
    "target_version" : "4.1.2.6.0.24723849",
    "upgrade_bundle_file_name" : "VMware-NSX-upgrade-bundle-4.1.2.6.0.24723849-pre-check.pub",
    "component_target_versions" : [ {
      "component_type" : "EDGE"
    }, {
      "component_type" : "HOST"
    }, {
      "component_type" : "MP"
    } ],
    "upgrade_coordinator_updated" : true
  }
  

• A PUB file upload in NSX manager will be reported in the log file, /var/log/upgrade-coordinator/upgrade-coordinator.log 
  

  YYYY-MM-DDT23:06:37.093Z  INFO http-nio-127.0.0.1-7442-exec-3 UpgradeCoordinatorFacadeImpl 3097 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Upload upgrade bundle VMware-NSX-upgrade-bundle-4.1.2.6.0.24723849-pre-check.pub via file and verifying in Async with install flag true
  ...
  YYYY-MM-DDT23:07:09.797Z  INFO Thread-11 UpgradeCoordinatorUpgradeService 3097 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Since Bundle version is not available. This is fresh install case. Hence storing the version for UcUpgradeContext{upgradeBundleName='VMware-NSX-upgrade-bundle-4.1.2.6.0.24723849-pre-check.pub', upgradeBundleType=PUB, upgradeBundleVersion='4.1.2.6.0.24723849', activeUpgradeBundleService=com.vmware.nsx.management.upgrade.bundle.PUBUpgradeBundleService@446fb362, targetVersion='null', metadata=2, preUpgradeBundleMetadata={NSX_EDGENODE_VMDK_SIZE=3358831616, NSX_MANAGER_VMDK_SIZE=4421377536, NSX_MANAGER_NUB_SIZE=18995200, NSX_EDGENODE_NUB_SIZE=34529280}}

Environment

VMware Cloud Foundation 5.x

Cause

The SDDC manager precheck workflow breaks when the Pre-Upgrade Bundle(PUB) file is uploaded to the NSX Manager and a pre-check is done. 

The pre-check run on the NSX manager directly is considered an out-of-band activity and is not supported with VCF 5.x

Resolution

To work around the issue, 

Step 1: Backup SDDC Manager and NSX Managers

Step 2: Upgrade the upgrade coordinator

1. Download the MUB file - VMware-NSX-upgrade-bundle-4.1.2.6.0.24723849.mub
2. Login as local admin user to the NSX Manager
3. Select System -> Upgrade from the navigation panel.
4. Click Upgrade
5. Click Browse to navigate to the location you downloaded the upgrade bundle .mub file.
6. Click Upload.
7. When the upload process finishes, the Prepare for Upgrade button appears. Click Prepare for Upgrade to upgrade the upgrade coordinator.
8. Read and accept the EULA terms.
9. Accept the notification to upgrade the upgrade coordinator.
10. Click Run Pre-Checks to verify that all the NSX components are ready for upgrade.

Step 3: Validate if the component target versions are populated now.

curl -k -u admin -X GET -H "Content-Type:application/json" https://<NSX MANAGER VIP FQDN>/api/v1/upgrade/summary

Step 4: Validate prerequisites.

1. In the SDDC manager UI, go to developer center -> API Explorer
2. Validate -> upgradables -> Select Get next to /v1/upgradables/domains/{domainid}/nsxt
3. Enter the domainid value for the domain and select execute.
4. Validate that the following components are healthy - NsxtEdgeCluster, NsxtHostCluster, NsxtManagerCluster and NsxtUpgradeCoordinator.
5. Ideally, resourceHealth should be UP/STABLE/HEALTHY and Upgrade Coordinator upgrade status marked as COMPLETED

Step 5: Run prechecks through SDDC-M and complete the upgrade.