After upgrading NSX prepared ESXi host transport nodes from ESXi 7.0 to 8.0, NSX UI shows the hosts "Failed to Install Error"
search cancel

After upgrading NSX prepared ESXi host transport nodes from ESXi 7.0 to 8.0, NSX UI shows the hosts "Failed to Install Error"

book

Article ID: 378075

calendar_today

Updated On:

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

After upgrading ESXi host from 7.0 to 8.0, the NSX prepared host cluster shows the hosts are in 'Failed' state.

Host goes into 'Install Failed' state with error 'Node has invalid version <vib verision> of <vib name>'.

Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-monitoring.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-vdpi.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-shared-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proxy.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-netopa.Software nsx-snproxy not present on host.Node has invalid version 0.4.14.py35-19345965 of software nsx-python-greenlet.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-utils.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-nestdb.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-esx-datapath.Node has invalid version 1.3.5.py35-19972216 of software nsx-python-gevent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-logging.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-context-mux.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-exporter.Node has invalid version 4.1.1.0.0-7.0.22224315 of software vsipfwlib.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-ids.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-sfhc.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-opsagent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsxcli.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cpp-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proto2-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-adf.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-platform-client.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cfgagent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-host.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-mpa.

Error message seen on NSX Manager UI in NSX Configuration Status :

Points to Note:

  • If looking at the invalid version of the NSX VIBs, an example invalid version is: 4.1.1.0.0-7.0.22224315
  • Datapath of NSX workloads may not be impacted right after the ESXi upgrade, i.e., VMs connected to a NSX managed segment still have network access.
  • VMs may not be able to migrate to nodes affected by this issue with related tasks in vCenter showing failure messages as below:
    11/01/2024, 6:30:00 PM The operation performed on <host name> in Datacenter timed out
    11/01/2024, 5:30:00 PM Unable to automatically migrate <vm name> from <host name>

Environment

NSX 3.x, 4.x

vSphere ESXi 7.0, 8.0

 

Cause

The newly upgraded hosts do not have correct NSX VIBs and so the Install status validation fails resulting in 'Failed Install' status. The NSX VIBs are different for ESXi 7.0 and ESXi 8.0.

Resolution

In a VMware NSX setup, before an upgrade to ESXi 8.0 and later, make sure that the NSX kernel module is part of the image or upgrade baseline.
For any hosts that are not upgraded, follow this document: Upgrading ESXi Hosts in an Environment With VMware NSX

For hosts that were upgraded and showing errors:

  1. Navigate in the NSX UI to Systems > Fabric > Nodes > Host Transport Nodes (NSX-T 3.2.1 or prior)
    • Systems > Fabric > Hosts > Clusters (NSX-T 3.2.2 and newer). 
  2. Click on the "Failed" state of the host error message
  3. Click on the specific step that fails
  4. Select the error message and click on "Resolve". This should trigger a re-installation of NSX VIBs on the newly upgraded ESXi 8.0 hosts and soon after, the status of the hosts will show "up". 
  5. Expect the process to take approximately 10 minutes if it works successfully, much of which will show it at 18% in the UI. If it still fails to complete, it is recommended to place the host into Maintenance Mode and reboot it.

If the above workarounds do not remediate the issues, please open a Broadcom Support Request and reference this KB article. Also upload all the 3 NSX Managers and ESXi host support bundles to the case along with details such as timestamps, names of impacted areas (hostnames, etc.), and the text of any other error messages seen. 

Powered by Wolken Software