ESXi Host shows a status of NSX INSTALL FAILED state with error "Node has invalid version <vib version> of <vib name>" after updating ESXi

Products

VMware NSX

Issue/Introduction

After updating ESXi, the "NSX Configuration" status shows as "Install Failed" in the NSX Manager when viewing the host's (Transport Node) status:

When clicking on "Install Failed", the following errors are reported (the specific version mentioned in the error will vary based on the NSX version installed):

Node has invalid version 4.2.1.0.0-7.0.24302014 of software nsx-monitoring.Node has invalid version 4.2.1.0.0-7.0.24302014 of software nsx-vdpi.Node has invalid version 4.2.1.0.0-7.0.24302014 of software nsx-shared-libs.Node has invalid version 4.2.1.0.0-7.0.24302014 of software nsx-proxy.Node ha...

The behavior is similar to that seen in the following KB, however the error is with the "Installing NSX" stage and not "Configuration complete":

ESXi Host goes into NSX INSTALL FAILED state with error "Node has invalid version <vib version> of <vib name>" post upgrade of ESX host OS and NSX vibs together

Clicking "RESOLVE" in the error details does not resolve the error.

When viewing the NSX VIBs on the ESXi host in question, the VIB versions do not match the currently installed ESXi version. You can see the NSX VIBs installed with the following command on the ESXi host:

# esxcli software vib list | grep nsx

For example if the host is running ESXi 8.0, you may see the NSX VIBs running a version ending in 7.0 indicating the VIB has not been updated with the ESXi update:

You may see the following messages in the esxupdate log file located in /var/run/log on the host:

YYYY-MM-DDTHH:MM:SSZ In(14) esxupdate[#######] Running [/etc/init.d/nsx-pre-cfgagent start upgrade]…

YYYY-MM-DDTHH:MM:SSZ In(14) esxupdate[#######] runcommand called with: args = ['/etc/init.d/nsx-pre-cfgagent', 'start', 'upgrade'], outfile = None, returnoutput = True, timeout = 0.

YYYY-MM-DDTHH:MM:SSZ Wa(12) esxupdate[#######] Error in running [/etc/init.d/nsx-pre-cfgagent start upgrade]:

YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] esxupdate[#######] Return code: 1

YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] esxupdate[#######] Output: /tmp/nsx2/.nsx_dp_upgrade_in_progress exists, returning

YYYY-MM-DDTHH:MM:SSZ In(14) esxupdate[#######] Running [/etc/init.d/nsx-pre-idps start upgrade]...

YYYY-MM-DDTHH:MM:SSZ In(14) esxupdate[#######] runcommand called with: args = ['/etc/init.d/nsx-pre-idps', 'start', 'upgrade'], outfile = None, returnoutput = True, timeout = 0.

YYYY-MM-DDTHH:MM:SSZ Wa(12) esxupdate[#######] Error in running [/etc/init.d/nsx-pre-idps start upgrade]:

YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] esxupdate[#######] Return code: 1

YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] esxupdate[#######] Output: /tmp/nsx2/.nsx_dp_upgrade_in_progress exists, returning

Environment

VMware NSX 4.x

Cause

This behavior is seen when the NSX VIBs on the ESXi host are not the correct version for the ESXi host itself. This mismatch can generally be remedied with the "RESOLVE" workflow as detailed in the below KB, but sometimes other issues can prevent the VIBs from being correctly installed.

ESXi Host goes into NSX INSTALL FAILED state with error "Node has invalid version <vib version> of <vib name>" post upgrade of ESX host OS and NSX vibs together

Resolution

There are two methods available to resolve this issue:

Option 1: Reinstall NSX VIBs by Removing/Re-Adding Host to Cluster

Place the host into Maintenance Mode (MM).
Move the host out of the cluster to the Datacenter level.
In the NSX UI, go to System > Fabric > Hosts > Other Nodes and confirm the host shows as Not Configured.
- If the host does not show as Not Configured, select the host and click REMOVE NSX.
Once the host shows as Not Configured, SSH to the host and run the following to confirm no NSX VIBs remain:
```
esxcli software vib list | grep nsx
```
Move the host back into the cluster. The correct version of NSX VIBs should install automatically.

Option 2: Manually Install the Correct VIB (No-Live Method)

Download the correct VIB package from the Broadcom Customer Portal.
- Example file format:
```
nsx-lcp-4.2.1.0.0.24302014-esx80.zip
```
Reboot the host to clear any incomplete upgrade tasks.
Copy the downloaded NSX VIB package to the host’s /tmp directory using an SCP client.
SSH into the host as root and install the VIB with the following command (update filename as needed):
```
esxcli software vib install --no-live-install -d /tmp/nsx-lcp-4.2.1.0.0.24302014-esx80.zip
```
Reboot the ESXi host again.
After reboot, verify the correct VIB is installed:
```
esxcli software vib list | grep nsx
```
If needed, return to the NSX Manager and click Resolve on the host to sync version information. Once synced, the host should show a status of Success.

Additional Information

The error messages in the esxupdate log are identical to those detailed in the following KB, requiring the same workaround:

NSX-T host upgrade is failing on hosts with error "File "/etc/init.d/nsx-datapath-dl", <module> DualLoadUpgrade()"