ESXi Host goes into NSX INSTALL FAILED state with error "Node has invalid version <vib verision> of <vib name>" post upgrade of ESX host OS and NSX vibs together

Products

• VMware NSX

Issue/Introduction

User upgrades ESX host OS and NSX vibs together and Host goes into INSTALL FAILED state with error "Node has invalid version <vib verision> of <vib name>" for all bulletin ids of the Host.

Install Failed Error :

Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-monitoring.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-vdpi.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-shared-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proxy.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-netopa.Software nsx-snproxy not present on host.Node has invalid version 0.4.14.py35-19345965 of software nsx-python-greenlet.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-utils.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-nestdb.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-esx-datapath.Node has invalid version 1.3.5.py35-19972216 of software nsx-python-gevent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-logging.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-context-mux.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-exporter.Node has invalid version 4.1.1.0.0-7.0.22224315 of software vsipfwlib.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-ids.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-sfhc.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-opsagent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsxcli.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cpp-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proto2-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-adf.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-platform-client.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cfgagent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-host.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-mpa.

Error message seen on NSX Manager UI in NSX Configuration Status :

Environment

NSX Data Center 3.x

NSX 4.x

Cause

This is a corner case with timing issue as the root cause.

Issue happens sometimes after OS upgrade. When a host is rebooted, NSX MP gets MT_INIT event from the host. MP responds to this event with required VIBs list. But there is a state API call that happens sometimes before the above event and till this point host has the old bulletin list (related to old OS version). So when API is called, it shows version mismatch. Had the state API been called a minute later, we will not see the issue.

Resolution

After host OS upgrade is completed and host is successfully rebooted, follow the resolver workflow. This is same workflow which is followed when host TN creation fails:

Put the Host in MM and on NSX UI, click on "Install Failed" status of the host. In popup will show the above error message. Select the error and click on resolve.

Additional Information

If a user tries to monitor the status of OS upgrade through automation, then there is a chance of incorrect reporting, but the resolver workflow sorts the issue out.

The signature for this issue is the same for issue identified in KB: Using VLCM to upgrade vSphere and NSX it fails with "Node has invalid version 4.2.1.0.0-8.0.24302041 of software nsx-monitoring..."

The cause of this issue is different but the error message is the same.