After upgrading NSX prepared ESXi host transport nodes from ESXi 7.0 to 8.0, NSX UI shows the hosts "Failed to Install Error"
search cancel

After upgrading NSX prepared ESXi host transport nodes from ESXi 7.0 to 8.0, NSX UI shows the hosts "Failed to Install Error"

book

Article ID: 378075

calendar_today

Updated On:

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

After upgrade ESXi host from 7.0 to 8.0, the NSX prepared host cluster shows the hosts were in "failed" state.

Host goes into INSTALL FAILED state with error "Node has invalid version <vib verision> of <vib name>".

Install Failed Error :

Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-monitoring.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-vdpi.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-shared-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proxy.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-netopa.Software nsx-snproxy not present on host.Node has invalid version 0.4.14.py35-19345965 of software nsx-python-greenlet.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-utils.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-nestdb.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-esx-datapath.Node has invalid version 1.3.5.py35-19972216 of software nsx-python-gevent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-python-logging.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-context-mux.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-exporter.Node has invalid version 4.1.1.0.0-7.0.22224315 of software vsipfwlib.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-ids.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-sfhc.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-opsagent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsxcli.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cpp-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-proto2-libs.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-adf.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-platform-client.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-cfgagent.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-host.Node has invalid version 4.1.1.0.0-7.0.22224315 of software nsx-mpa.

Error message seen on NSX Manager UI in NSX Configuration Status :

Points to Note:

  • If looking at the invalid version of the NSX VIBs, an example invalid version is: 4.1.1.0.0-7.0.22224315
  • There is no immediate impact of this issue, VMs connected to a NSX managed segment still have network access.
  • VMs may not be able to migrate to nodes affected by this issue with related tasks in vCenter showing failure messages such as:

11/01/2024, 6:30:00 PM The operation performed on <host name> in Datacenter timed out
11/01/2024, 5:30:00 PM Unable to automatically migrate <vm name> from <host name>

Environment

NSX-T Datacenter 3.x

NSX 4.x

vSphere ESXi 7.0

vSphere ESXi 8.0

 

Cause

This is caused by a missing step in host upgrade preparation, the NSX kernel module for the appropriate ESXi version that the hosts are upgraded to. 

The newly upgraded hosts do not have appropriate NSX VIBs thus the connection to NSX manager resulting in failure. 

Resolution

In a VMware NSX setup, before an upgrade to ESXi 8.0 and later, make sure that the NSX kernel module is part of the image or upgrade baseline.
For any hosts that are not upgraded, follow this document: Upgrading ESXi Hosts in an Environment With VMware NSX 

For hosts that were upgraded and showing errors:

  1. Navigate in the NSX UI to Systems > Fabric > Nodes > Host Transport Nodes (NSX-T 3.2.1 or prior)
    • Systems > Fabric > Hosts > Clusters (NSX-T 3.2.2 and newer). 
  2. Click on the "failed" state of the host error message
  3. Click on the specific step that fails
  4. Select the error message and click on "resolve". This should trigger a re-installation of NSX VIBs on the newly upgraded ESXi 8.0 hosts and soon after, the status of the hosts will show "up". 
  5. Expect the process to take approximately 10 minutes if it works successfully, much of which will show it at 18% in the UI. If it still fails to complete, it is recommended to place the host into Maintenance Mode and reboot it.

If the above workarounds do not remediate the issues, please open a Broadcom Support Request and reference this KB article. Also upload all the 3 NSX Managers and ESXi host support bundles to the case along with details such as timestamps, names of impacted areas (hostnames, etc.), and the text of any other error messages seen. 

Powered by Wolken Software