NSX Upgrade vLCM managed hosts fails with "Set desired state operation failed. Image is not valid" or "has unmet dependencies"

Products

VMware SDDC Manager VMware NSX

Issue/Introduction

In VMware NSX 4.x, when attempting to upgrade NSX with host clusters managed by vLCM, the host upgrade may fail due to "unmet dependencies" error.

If you are upgrading a workload domain that uses vSphere Lifecycle Manager images and its cluster image was created from an ESXi host that uses vSphere Lifecycle Manager baselines, the NSX host upgrade may fail.

Similar log entries may be found in NSX manager:

/var/log/upgrade-coordinator/upgrade-coordinator.log

2025-01-16T06:18:44.904Z  INFO task-executor-43-1-workitem-HOST-vlcm-<UUID> VLCMOperationsServiceImpl 561175 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] getTaskStatus response part-1: {"value":{"parent":"","cancelable":true,"subtasks":[{"value":{"parent":"<UUID>","cancelable":true,"end_time":"2025-01-16T06:14:23.290Z","description":{"args":[],"default_message":"Task created by VMware vSphere Lifecycle Manager","localized":"Task created by VMware vSphere Lifecycle Manager","id":"com.vmware.vcIntegrity.lifecycle.Task.Description"},"target":{"id":"<ID>","type":"ClusterComputeResource"},"result":{"incompatible_hosts":[],"hosts":[{"value":{"components":[],"solutions":[{"value":{"current":{"components":[{"component":"nsx-lcp-bundle"}],"details":{"components":[{"component":"nsx-lcp-bundle","display_version":"NSX LCP Bundle(4.1.2.5.0-8.0.24150844)","vendor":"VMware","display_name":"NSX LCP Bundle"}],"display_version":"4.1.2.5.0","display_name":"VMware NSX-T"},"version":"4.1.2.5.0-24150844"},"stage_status":"NOT_STAGED","notifications":{},"status":"NON_COMPLIANT","target":{"components":[{"component":"nsx-lcp-bundle"}],"details":{"components":[{"component":"nsx-lcp-bundle","display_version":"NSX LCP Bundle(4.2.1.1.0-8.0.24405894)","vendor":"VMware","display_name":"NSX LCP Bundle"}],"display_version":"4.2.1.1.0","display_name":"VMware NSX-T"},"version":"4.2.1.1.0-24405894"}}......

vCenter logs may also show:

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

"id": "com.vmware.vcIntegrity.lifecycle.Validate.ProviderComponentObsoleted",^M
-->                     "message": {^M
-->                         "args": [^M
-->                             "NSX LCP Bundle(NSX LCP Bundle(4.1.2.1.0-8.0.22667792))",^M
-->                             "nsx-python-gevent-esxio",^M
-->                             "NSX LCP Bundle(NSX LCP Bundle(4.1.2.1.0-8.0.22667792))"^M
-->                         ],^M
-->                         "default_message": "Component {1} has unmet dependency {2} because providing component(s) {3} are obsoleted.",^M

The unmet dependencies can be one or more of the following (not an exhaustive list):

nsx-python-gevent
nsx-python-gevent-esxi
nsx-python-greenlet
nsx-python-greenlet-esxio

Environment

VMware Cloud Foundation 5.2

VMware NSX 4.x

Cause

When features like HA, NSX, or WCP are enabled in a Baseline Cluster, and a user attempts to seed an image from a host in that cluster, the process treats these features as regular components instead of specialized solutions. This occurs because Baseline Clusters lack the metadata needed to recognize them as solutions, a concept specific to vLCM Clusters.
If an image created from such a Baseline Cluster is used to set up a new cluster, and an NSX upgrade is later performed on the hosts in that cluster, the upgrade may fail due to conflicts with the nsx-lcp-bundle component.

A cluster image created from an ESXi host that uses vSphere Lifecycle Manager baselines contains an NSX component that causes this issue.

Resolution

Issue resolved in ESXi and vCenter 8.0U3 or later, however, if a cluster was created prior to this, this issue may still persist in the environment and a workaround is required to remediate this issue.

Workaround:

Do not create cluster images from an ESXi host that uses vSphere Lifecycle Manager baselines. If you encounter this issue, you can resolve it by using the vSphere Client to remove the NSX LCP Bundle component from the cluster image.

This change is a per cluster change and needs to be done for every cluster in an affected SDDC manager environment.

If the above steps in the UI are not working. Try the following steps:

Log in to vSphere Client
Navigate to affected cluster
Select "Updates" tab
Export cluster image as JSON
Edit JSON file
In the "components" section, remove the entire line that contains "nsx-lcp-bundle"
Save the modified JSON file
Import the modified JSON file
Verify and save the updated cluster image
Retry NSX upgrade in SDDC Manager

This change will correct the compliance check without altering the configuration on the host or affecting running VMs.

Additional Information

VCF 5.2 Release Notes Known Issues