Error: Failed to execute the resource validations during NSX Manager upgrade in VCF

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

When attempting to upgrade NSX Managers from 9.0.2 to 9.1.0100 within a VMware Cloud Foundation (VCF) environment:

• The upgrade precheck fails with the error message: Failed to execute the resource validations.
  • 
• Upgrade tasks in the SDDC Manager UI remain in a Pending state and do not progress, even after several hours.
• The NSX Manager Upgrade Coordinator UI may show the precheck as running indefinitely, or the Run Pre-check button may be greyed out.
• Lifecycle Management (LCM) logs in SDDC Manager (/var/log/vmware/vcf/lcm/vcf-lcm.log) may indicate timeouts connecting to the NSX Upgrade Coordinator.
• lcm-debug.log in SDDC Manager may contain logged message: "NSX_T_MANAGER target verson not found in TargetProductVersions. Falling back to LCM manifest for release: 9.0.2.0.
  • 

Environment

VMware Cloud Foundation 9.x

NSX 9.x

SDDC 9.x

Cause

This issue occurs due to a stale internal state within the NSX Upgrade Coordinator and/or a lock in the SDDC Manager orchestration engine.

A) NSX stale state: The NSX Manager upgrade plan metadata can become inconsistent or corrupted in the NSX database, preventing new validations from executing. Additionally, JVM deadlocks or hung threads within the upgrade-coordinator service can cause the daemon to become unresponsive to orchestration calls.

B) SDDC Manager lock: The Lifecycle Management (LCM) service may hold a stale orchestration lock on the upgrade task, preventing the task from failing over or retrying correctly.

Resolution

To resolve the locked state and allow the prechecks to complete, follow the steps below to reset both the service daemons and the persisted upgrade metadata.

1. Restart the upgrade-coordinator service on all NSX nodes.
   1. SSH into each NSX Manager node as root.
   2. Restart the local upgrade coordinator service to clear out any frozen validation threads:
      systemctl restart upgrade-coordinator   -or-   /etc/init.d/upgrade-coordinator restart
   3. Optional: If the service restart doesn't fully clear the state, you can manually reset the management plane upgrade plan on the NSX primary node via the following API call:
      curl -ku 'admin' -X POST 'https://<NSX_MANAGER_IP>/api/v1/upgrade/plan?action=reset&component_type=MP'
      
2. Restart the lcm service on SDDC Manager to clear the stuck "Pending" status.
   1. SSH into the SDDC Manager as the vcf user.
   2. Switch to the root user with su -
   3. Restart the lifecycle manager service by running systemctl restart lcm
3. Wait about 10 minutes for the internal APIs to stabilize, then trigger the precheck again from the VCF interface.

Additional Information

Commands and Calls explained

• curl -ku 'admin' -X POST 'https://<NSX_MANAGER_IP>/api/v1/upgrade/plan?action=reset&component_type=MP'   -- API call to NSX to reset the upgrade plane state for the mgmt plane. This does not reset the mgmt plane itself.
• systemctl restart upgrade-coordinator   -- Restarts the background service (the daemon) running on the Linux operating system of the NSX Manager. This clears the active RAM, drops any hung threads, and forces the Java process running the upgrade coordinator to spin down and start fresh. This does not delete the saved configuration, or the persisted state of the upgrade plan stored in the NSX database. To clear the upgrade plan, the API call above must be used.
• /etc/init.d/upgrade-coordinator restart   -- This command the exact same thing as systemctl restart.
• systemctl restart lcm   -- This triggers an orchestration flush on SDDC LCM. Clears stuck upgrade tasks.

 

For more information, Contact Broadcom support.

Refer to:

• The NSX upgrade tab never finishes loading and displays a spinning circle indefinitely after NSX Manager node is re-deployed mid-upgrade
•  
• NSX upgrade on VCF fails with "Read timed out, httpStatus : , error_code : 0"