After upgrade ESXi host from 7.0 to 8.0, the NSX prepared host cluster shows the hosts were in "failed" state.
Host goes into INSTALL FAILED state with error "Node has invalid version <vib verision> of <vib name>".
Install Failed Error :
Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-monitoring.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-vdpi.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-shared-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proxy.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-netopa.Software nsx-snproxy not present on host.Node has invalid version 0.4.14.py35-19345965 of software nsx-python-greenlet.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-utils.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-nestdb.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-esx-datapath.Node has invalid version 1.3.5.py35-19972216 of software nsx-python-gevent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-logging.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-context-mux.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-exporter.Node has invalid version 4.1.1.0.0-7.0.22224315 of software vsipfwlib.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-ids.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-sfhc.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-opsagent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsxcli.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cpp-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proto2-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-adf.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-platform-client.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cfgagent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-host.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-mpa.
Error message seen on NSX Manager UI in NSX Configuration Status :
Points to Note:
11/01/2024, 6:30:00 PM The operation performed on <host name> in Datacenter timed out
11/01/2024, 5:30:00 PM Unable to automatically migrate <vm name> from <host name>
NSX-T Datacenter 3.x
NSX 4.x
vSphere ESXi 7.0
vSphere ESXi 8.0
This is caused by a missing step in host upgrade preparation, the NSX kernel module for the appropriate ESXi version that the hosts are upgraded to.
The newly upgraded hosts do not have appropriate NSX VIBs thus the connection to NSX manager resulting in failure.
In a VMware NSX setup, before an upgrade to ESXi 8.0 and later, make sure that the NSX kernel module is part of the image or upgrade baseline.
For any hosts that are not upgraded, follow this document: Upgrading ESXi Hosts in an Environment With VMware NSX
For hosts that were upgraded and showing errors:
If the above workarounds do not remediate the issues, please open a Broadcom Support Request and reference this KB article. Also upload all the 3 NSX Managers and ESXi host support bundles to the case along with details such as timestamps, names of impacted areas (hostnames, etc.), and the text of any other error messages seen.