ESXi Host Scan error after transitioning NSX VUM cluster to vLCM with error on running Compliance check: "An unknown error occurred while performing the operation"

Products

• VMware NSX
• VMware vSphere ESX 8.x

Issue/Introduction

Symptoms

• Customer is trying to convert an NSX-T enabled cluster from using VUM Baselines to vLCM Images and hitting this KB: Switching from using Baselines to Images (vLCM) may result in drift and report VIB/Component Removal Notifications
• As per the resolution mentioned in KB, customer goes ahead with conversion but once that completes, all Hosts in the Cluster show up in "Unknown" state and unable to complete Compliance checks with the following error: "An unknown error occurred while performing the operation."

Logs

• Log file /var/run/log/lifecycle.log on ESXi Host will show similar to below snippets:

2024-07-12T20:08:43Z Er(11)[+] lifecycle[2235237]: Cannot merge unequal components with id 'vmw-nsx-lcp_4.0.1.1.0.20598730-esx80-snic-system' and 'vmw-nsx-lcp_4.0.1.1.0.20598730-esx80'
2024-07-12T20:08:43Z Er(11) lifecycle[2235237]: imagemanagerctl:101 ValueError: Cannot merge unequal components with id 'vmw-nsx-lcp_4.0.1.1.0.20598730-esx80-snic-system' and 'vmw-nsx-lcp_4.0.1.1.0.20598730-esx80'

 

• Update Manager log on vCenter Server /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log will log similar to below snippets:

2024-07-12T20:09:13.547Z info vmware-vum-server[2902291] [Originator@6876 sub=ScanUtil] [ScanUtil 1159] Successfully retrieved software spec for the entity : ####-G-DEV-####-01 (domain-c####)
  {
     "add_on": {
         "name": "DEL-ESXi",
         "version": "802.23305546-A07"
     },
     "base_image": {
         "version": "8.0.2-0.25.23305545"
     },
     "components": null,
     "hardware_support": null,
     "solutions": {
         "com.vmware.nsxt": {
             "components": [
                 {
                     "component": "nsx-lcp-bundle"
                 }
             ],
             "version": "4.0.1.1.0-20598730"
         },
         "com.vmware.vsphere-ha": {
             "components": [
                 {
                     "component": "vsphere-fdm"
                 }
             ],
             "version": "8.0.2-22617221"
         }
     }
 }

Environment

NSX 4.x

VMware vSphere ESXi 8.x

Cause

• There is a discrepancy in vmw-nsx-lcp component installed on the ESXi Host (localcli_software-component-get.txt):

vmw-nsx-lcp_4.0.1.1.0.20598730-esx80:
   Name: nsx-lcp-bundle
   Display Name: NSX LCP Bundle
   Version: 4.0.1.1.0-8.0.20598730
   Display Version: NSX LCP Bundle(4.0.1.1.0-8.0.20598730)

and the vmw-nsx-lcp component being pushed by vLCM/NSX-MP (present under /repository/<MP version>/HostComponents/esx80):

vmw-nsx-lcp_4.0.1.1.0.20598730-esx80-snic-system:
   Name: nsx-lcp-bundle
   Display Name: NSX LCP Bundle
   Version: 4.0.1.1.0-8.0.20598730
   Display Version: NSX LCP Bundle(4.0.1.1.0-8.0.20598730)

• The discrepancy exists because the Hosts were manually prepared for NSX using an offline depot

Resolution

Re-prepare the ESXi Hosts/Cluster for NSX (delete and recreate TN which will push the correct nsx-lcp bundle to the host)