ESX Host shows a status of "Install Failed" or "Partial Success" state with error "Node has invalid version <vib version> of <vib name>" after updating ESX

Products

VMware NSX

Issue/Introduction

After updating ESXi, the "NSX Configuration" status shows as "Install Failed" in the NSX Manager when viewing the host's (Transport Node) status:
When clicking on "Install Failed", the following errors are reported (the specific version mentioned in the error will vary based on the NSX version installed):
Node has invalid version 4.2.1.0.0-7.0.24302014 of software nsx-monitoring.Node has invalid version 4.2.1.0.0-7.0.24302014 of software nsx-vdpi.Node has invalid version 4.2.1.0.0-7.0.24302014 of software nsx-shared-libs.Node has invalid version 4.2.1.0.0-7.0.24302014 of software nsx-proxy.Node ha...
The behavior is similar to that seen in the following KB, however the error is with the "Installing NSX" stage and not "Configuration complete":
ESXi Host goes into NSX INSTALL FAILED state with error "Node has invalid version <vib version> of <vib name>" post upgrade of ESX host OS and NSX vibs together
Clicking "RESOLVE" in the error details does not resolve the error.
When viewing the NSX VIBs on the ESXi host in question, the VIB versions do not match the currently installed ESX version. You can see the NSX VIBs installed with the following command on the ESX host:
esxcli software vib list | grep nsx
For example, if the host is running ESX 8.0, you may see the NSX VIBs running a version ending in 7.0 indicating the VIB has not been updated with the ESX update:
You may see the following messages in the esxupdate log file located in /var/run/log on the host:
YYYY-MM-DDTHH:MM:SSZ In(14) esxupdate[#######] Running [/etc/init.d/nsx-pre-cfgagent start upgrade]…
YYYY-MM-DDTHH:MM:SSZ In(14) esxupdate[#######] runcommand called with: args = ['/etc/init.d/nsx-pre-cfgagent', 'start', 'upgrade'], outfile = None, returnoutput = True, timeout = 0.
YYYY-MM-DDTHH:MM:SSZ Wa(12) esxupdate[#######] Error in running [/etc/init.d/nsx-pre-cfgagent start upgrade]:
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] esxupdate[#######] Return code: 1
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] esxupdate[#######] Output: /tmp/nsx2/.nsx_dp_upgrade_in_progress exists, returning
YYYY-MM-DDTHH:MM:SSZ In(14) esxupdate[#######] Running [/etc/init.d/nsx-pre-idps start upgrade]...
YYYY-MM-DDTHH:MM:SSZ In(14) esxupdate[#######] runcommand called with: args = ['/etc/init.d/nsx-pre-idps', 'start', 'upgrade'], outfile = None, returnoutput = True, timeout = 0.
YYYY-MM-DDTHH:MM:SSZ Wa(12) esxupdate[#######] Error in running [/etc/init.d/nsx-pre-idps start upgrade]:
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] esxupdate[#######] Return code: 1
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] esxupdate[#######] Output: /tmp/nsx2/.nsx_dp_upgrade_in_progress exists, returning

Environment

VMware NSX
VMware NSX-T Datacenter

Cause

The ESXi host has been updated manually, either via Update Manager (Baselines) or using an ESX ISO. These update methods do not upgrade the NSX VIBs installed on the host. The end result is an ESX host of a version newer than the NSX VIBs. This causes the "Invalid version" alert.

Resolution

There are two methods available to resolve this issue:

Option 1: Reinstall NSX VIBs by Removing/Re-Adding Host to Cluster

Place the host into Maintenance Mode (MM).
Move the host out of the cluster to the Datacenter level.
In the NSX UI, go to System > Fabric > Hosts > Other Nodes and confirm the host shows as Not Configured.
- If the host does not show as Not Configured, select the host and click REMOVE NSX.
Once the host shows as Not Configured, SSH to the host and run the following to confirm no NSX VIBs remain:
```
esxcli software vib list | grep nsx
```
Move the host back into the cluster. The correct version of NSX VIBs should install automatically.

Option 2: Manually Install the Correct VIB (No-Live Method)

Download the correct VIB package from the Broadcom Customer Portal.
- Example file format:
```
nsx-lcp-4.2.1.0.0.24302014-esx80.zip
```
Reboot the host to clear any incomplete upgrade tasks.
Copy the downloaded NSX VIB package to the host’s /tmp directory using an SCP client.
SSH into the host as root and install the VIB with the following command (update filename as needed):
```
esxcli software vib install --no-live-install -d /tmp/nsx-lcp-4.2.1.0.0.24302014-esx80.zip
```
Reboot the ESXi host again.
After reboot, verify the correct VIB is installed:
```
esxcli software vib list | grep nsx
```
If needed, return to the NSX Manager and click Resolve on the host to sync version information. Once synced, the host should show a status of Success.

Additional Information

The error messages in the esxupdate log are identical to those detailed in the following KB, requiring the same workaround:

NSX-T host upgrade is failing on hosts with error "File "/etc/init.d/nsx-datapath-dl", <module> DualLoadUpgrade()"

ESXi Host goes into NSX INSTALL FAILED state with error "Node has invalid version <vib version> of <vib name>" post upgrade of ESX host OS and NSX vibs together