Error: "Node has invalid version <vib version> of <vib name>" encountered during simultaneous upgrades via NSX Manager and vCenter vLCM
search cancel

Error: "Node has invalid version <vib version> of <vib name>" encountered during simultaneous upgrades via NSX Manager and vCenter vLCM

book

Article ID: 413104

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

This issue occurs specifically when a simultaneous upgrade is in progress on both the NSX components and the ESXi hosts.

  • The NSX upgrade process from 4.1.x to 4.2.x is currently paused at the host upgrade stage from the NSX UI.
  • The Hosts within the vCenter cluster environment are already prepared for NSX via vLCM.
  • ESXi hosts were upgraded from version 7.x to 8.x using a vLCM image through vCenter. The vLCM image applied to the cluster includes the 'NSX LCP bundle (version 4.1.0.2.0.xxxx)' by default.

  • Moving the host to newly prepared vLCM cluster or non vLCM cluster do not report this issue. NSX install will be successful.

 

From the NSX UI, on the corresponding Transport Node Host, the following error Information reports with an option to resolve:

Node nas invalid version 4.1.0.2.0-7.0.21761693 of software nsx-monitoring.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-vdpl.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-shared-libs.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-proxy Node has invalid version 4.1.0.2.0-7.0 21761693 of software nsx-netopa.Software nsx-snproxy not present on host.Node has invalid version 0.4.14.py35-19345965 of software nsx-python-greenlet Node has invalid version 4.1.0.2.0-7 0.21761693 of software nsx-python-utils.Node has invalld version 4.1.0.2.0-7.0.21761693 of software nsx-nestdb.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-esx-datapath.Node has invalid version 1.3.5.py35-19972216 of software nsx-python-gevent.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-python-logging.Node has Invalid version 4.1.0.2.0-7.0.21761693 of software nsx-context-mux Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-exporter.Node has invalid version 4.1.0,2.0-7.0.21761693 of sortware vsiprwlib.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-ids.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-sfhc.Node has Invalid version 4.1.0.2.0-7.0.21761693 of software nsx-opsagent.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsxcli.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-cpp-libs Node has Invalid version 4.1.0.2.0-7.0.21761693 of software nsx-oroto2-tbs.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-adt.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-platform-client.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-crgagent.Node has invalid version 4.1.0.2.0-7.0.21761693 of software nsx-host Node has invalid version 41.0.2.0-7.0.21761693 of software nsx-mpa.Failed to install software on host. AI NSX vibs were not found on the host in vLCM based host prep flow.
  • Error screen provides an option to resolve, post clicking on resolve, another resolver window will appear.
    • Username: root , Password: <ESXi host root user password>, Thumbprint: ESXi host certificate thumbprint. 

  • Post clicking on resolve, popup error window will disappear and host status will become install failed with a new error as follows.
Failed to install software on host. Solution apply failed because the vSphere Lifecycle image contains either a new ESXi version or a new addons version or new components. Please proceed to the vSphere Client Lifecycle Manager to update ESXi or addons or components along with the solution 'com.vmware.nsxt' Solution apply failed on host: <ESXi-HostName> The 'Pensando' component version 'Pensando(4.2.1.3.0-8.0.24533885)' was deleted from vSphere Lifecycle image. Please proceed to the vSphere Client Lifecycle Manager to delete the component and update the solution 'com.vmware.nsxt'Solution apply failed on host: '<ESXi-HostName>'

  • Resolve will eventually fails at vLCM remediation eighter at 15% or at 23%.
Failed to install software on host. All NSX vibs were not found on the host in vLCM based host prep flow.

  • Resolving the above error will report another error as below.
Failed to install software on host. The specified solution is cannot be found in depots. Host <ESXi-HostName> was not processed, the reason: 'The specified solution is cannot be found in depots.'

  • Resolving above error will lead to failure at vLCM remediation or with a partial success.
    • The errors shared above may get seen in different order.

Environment

VMWare NSX

VMWare ESXi

Cause

  • The issue is caused by the absence of the required NSX LCP bundle (4.2.1.3.0-8.xxxx ) in the vLCM image
  • Since NSX upgrade coordinator is upgraded to a newer version i.e 4.2.1.3, the NSX Host remediation requires NSX LCP bundle 4.2.1.3 in vLCM.
  • The current vLCM image includes NSX LCP bundle version 4.1.0-7.xxxx, whereas version 4.2.1.3.0-8.xxxx is expected during the Host remediation process

Resolution

Recommended Upgrade Procedure:

  • Avoid performing simultaneous upgrades on the ESXi Transport Nodes via NSX and vCenter vLCM cluster.

  • Once the NSX upgrade is initiated, complete the full upgrade process via the NSX Manager(Edge, Host TNs, manager) before upgrading ESXi hosts from vCenter.

  • After the NSX upgrade is complete (Edge, Host TNs, manager), proceed with upgrading the ESXi hosts from vCenter.

Understanding of NSX Host Transport Node Preparation with vLCM:

  • In a vLCM-prepared cluster, a group of hosts share a common vLCM image:
    • Only when the NSX Transport Node Host Cluster is newly prepared and a Transport Node Profile (TNP) is attached, the required NSX LCP bundle is automatically pushed to the vLCM image associated with the vLCM cluster.

To address the issue, follow the steps below

On an existing NSX Transport Node Host cluster, NSX LCP bundle will not be pushed from NSX to the vCenter vLCM cluster image automatically.
But when the NSX upgrade is initiated and upgrade coordinator is upgraded to latest NSX LCP VIB, Stage option will help to push the required VIB bundle to vLCM image.

  • Go to NSX Upgrade page--> click on Hosts --> Go to the specific cluster by editing the existing cluster group --> Select the option "Stage in vSphere Lifecycle Manager" and click on Stage.
  • Now vLCM will receives the required VIB’s from NSX.

    • After selecting the above option, Login to the vCenter GUI --> Select the specific vLCM cluster --> Click on Updates
    • Verify the Image on cluster to check the required LCP VIB bundle is available.


  • Put the host in maintenance (if not already), move the corresponding failed host ( with maintenance mode enabled) out of cluster and add it back to the cluster for re-initiating the host preparation automatically.
  • vLCM remediation will be successful.
  • Complete the upgrade for all the hosts from vCenter and complete the NSX upgrade.

Additional Information

If the issue is not resolved, please collect below information and open a technical support case with Broadcom Support for further investigation.

  • NSX version.
  • ESXi Version.
  • Error screenshots with error information.
  • NSX Manager log bundles & problematic ESXi host long bundle along with  vCenter log bundle.
Powered by Wolken Software