NSX Host Install Failed After ESXi 7.0 to 8.0 Upgrade: Node has invalid version of software'
search cancel

NSX Host Install Failed After ESXi 7.0 to 8.0 Upgrade: Node has invalid version of software'

book

Article ID: 378075

calendar_today

Updated On:

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

After upgrading ESXi host from 7.0 to 8.0, the NSX prepared host cluster shows the hosts are in 'Failed' state.

Host goes into 'Install Failed' state with error 'Node has invalid version <vib verision> of <vib name>'.

 

Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-monitoring.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-vdpi.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-shared-libs.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proxy.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-netopa.Software nsx-snproxy not present on host.
Node has invalid version 0.4.14.py35-19345965 of software nsx-python-greenlet.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-utils.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-nestdb.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-esx-datapath.
Node has invalid version 1.3.5.py35-19972216 of software nsx-python-gevent.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-logging.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-context-mux.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-exporter.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software vsipfwlib.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-ids.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-sfhc.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-opsagent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsxcli.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cpp-libs.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proto2-libs.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-adf.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-platform-client.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cfgagent.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-host.
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-mpa.

Error message seen on NSX Manager UI in NSX Configuration Status :

Points to Note:

  • If looking at the invalid version of the NSX VIBs, an example invalid version is: 4.1.1.0.0-7.0.22224315
  • Datapath of NSX workloads may not be impacted right after the ESXi upgrade, i.e., VMs connected to a NSX managed segment still have network access.
  • VMs may not be able to migrate to nodes affected by this issue with related tasks in vCenter showing failure messages as below:
    11/01/2024, 6:30:00 PM The operation performed on <host name> in Datacenter timed out
    11/01/2024, 5:30:00 PM Unable to automatically migrate <vm name> from <host name>

Environment

NSX 3.x, 4.x

vSphere ESXi 7.0, 8.0

 

Cause

The newly upgraded hosts do not have correct NSX VIBs and so the Install status validation fails resulting in 'Failed Install' status. The NSX VIBs are different for ESXi 7.0 and ESXi 8.0.

Resolution

In a VMware NSX setup, before an upgrade to ESXi 8.0 and later, make sure that the NSX kernel module is part of the image or upgrade baseline.
For any hosts that are not upgraded, follow this document: Upgrading ESXi Hosts in an Environment With VMware NSX

For hosts that were upgraded and showing errors:

  1. Navigate in the NSX UI to Systems > Fabric > Nodes > Host Transport Nodes (NSX-T 3.2.1 or prior)
    • Systems > Fabric > Hosts > Clusters (NSX-T 3.2.2 and newer). 
  2. Click on the "Failed" state of the host error message located in the Host Transport Nodes section.
  3. Click on the specific step that fails
  4. Select the specific error message associated with the failed VIB installation and click on the "Resolve" button'. This should trigger a re-installation of NSX VIBs on the newly upgraded ESXi 8.0 hosts and soon after, the status of the hosts will show "up". 
  5. Expect the process to take approximately 10 minutes if it works successfully, much of which will show it at 18% in the UI. If it still fails to complete, it is recommended to place the host into Maintenance Mode and reboot it.

If the above workarounds do not remediate the issues, please open a Broadcom Support Request and reference this KB article. Also upload all the 3 NSX Managers and ESXi host support bundles to the case along with details such as timestamps, names of impacted areas (hostnames, etc.), and the text of any other error messages seen.  

Additional Information

Impact/Risks:
NSXA agent is marked as down and VMs cannot vmotion to the affected ESXi.

Vmotion fails with below error:
Currently connected network interface" 'Network adapter 1' uses network 'DVSwitch[00 ## 16 ## ## 58 1e ff ## ## ## 89 5e ## ## ##] NSX port group [dvportgroup-########](nsxa down)', which is not accessible

Upgrade precheck failed with error:
Connectivity issue found between manager and transport node(s) XXXXXXXXX-XXXX-XXXX-XXXX-XXXX. The issue may affect the upgrade if left unresolved.

Powered by Wolken Software