"NSX Manager is in error state due to audit failure. Please run upgrade pre-checks before proceeding with upgrade" error when updating NSX Manager nodes via SDDC
search cancel

"NSX Manager is in error state due to audit failure. Please run upgrade pre-checks before proceeding with upgrade" error when updating NSX Manager nodes via SDDC

book

Article ID: 402202

calendar_today

Updated On:

Products

VMware SDDC Manager VMware NSX

Issue/Introduction

  • Updating/upgrading NSX-T Manager via SDDC UI fails with error:
    "NSX Manager is in error state due to audit failure. Please run upgrade pre-checks before proceeding with upgrade"

  • In /var/log/vmware/vcf/lcm/lcm-debug.log, the below error message is logged:
    Caused by: java.lang.RuntimeException: NSX Upgrade Coordinator could not be fetched

  • In /var/log/vmware/vcf/lcm/lcm-debug.log, the below error message is logged:
    Error auditing NSX CLuster <cluster name>  with exception {} com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSXZ CLuster from the Inventory

  • In the NSX Manager UI, there are more than 3 NSX-T Manager nodes deployed, but from SDDC Manager, only 3 are recognized.

Environment

VMware Cloud Foundation 5.x
VMware NSX

Cause

  • The SDDC Manager only supports deploying an NSX environment with exactly 3 NSX Manager nodes in an NSX-T cluster.
    • Deploying more than 3 NSX Managers via NSX Manager UI is not supported by SDDC Manager’s workflows.
    • If more appliances are manually added, it is not recognized by SDDC Manager and breaks the lifecycle operations.

  • 'Upgrade coordinator' is a built-in NSX Manager component that orchestrates the entire NSX-T upgrade workflow.
    • It runs as a service on all NSX Manager nodes, but only one node acts as the active Upgrade Coordinator at a time.
    • In such cases, if the 'Upgrade coordinator' is active on a node not recognized by the SDDC Manager, the update/upgrade task will fail with the above error message.

Resolution

  • Ensure only 3 NSX Manager nodes are deployed from the SDDC Manager.
  • Delete the remaining NSX Manager nodes deployed via NSX-T UI.

Workaround:

  1. Log into the NSX Manager VIP using the admin user credentials.
  2. Run the following command to obtain the IP address of the current orchestrator / repository node, and confirm that the IP address is from one of the NSX Managers not recognized by the SDDC Manager. 
    get service install-upgrade
  3. Set the NSX Manager VIP as the upgrade coordinator current orchestrator / repository node with the following command 
    set repository-ip
  4. Restart the lcm service 
    systemctl restart lcm

Go back to the SDDC Manager UI and run the Upgrade Pre-checks again, as they will now succeed

Additional Information

Powered by Wolken Software