ESXi Host Scan error after transitioning NSX VUM cluster to vLCM with error on running Compliance check: "An unknown error occurred while performing the operation"

Products

• VMware NSX
• VMware vSphere ESX 8.x

Issue/Introduction

• When trying to convert an NSX-T enabled cluster from using VUM Baselines to vLCM Images and hitting this KB: Switching from using Baselines to Images (vLCM) may result in drift and report VIB/Component Removal Notifications
• As per above KB after conversion completes, all Hosts in the Cluster show up in "Unknown" state and unable to complete Compliance checks with the following error: "An unknown error occurred while performing the operation."
• Log file /var/run/log/lifecycle.log on ESXi Host will show similar to below snippets:

Er(11)[+] lifecycle[2235237]: Cannot merge unequal components with id 'vmw-nsx-lcp_4.0.1.1.0.20598730-esx80-snic-system' and 'vmw-nsx-lcp_4.0.1.1.0.20598730-esx80'
Er(11) lifecycle[2235237]: imagemanagerctl:101 ValueError: Cannot merge unequal components with id 'vmw-nsx-lcp_4.0.1.1.0.20598730-esx80-snic-system' and 'vmw-nsx-lcp_4.0.1.1.0.20598730-esx80'

• Update Manager log on vCenter Server /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log will log similar to below snippets:

info vmware-vum-server[2902291] [Originator@6876 sub=ScanUtil] [ScanUtil 1159] Successfully retrieved software spec for the entity : ####-G-DEV-####-01 (domain-c####)
  {
     "add_on": {
         "name": "DEL-ESXi",
         "version": "802.23305546-A07"
     },
     "base_image": {
         "version": "8.0.2-0.25.23305545"
     },
     "components": null,
     "hardware_support": null,
     "solutions": {
         "com.vmware.nsxt": {
             "components": [
                 {
                     "component": "nsx-lcp-bundle"
                 }
             ],
             "version": "4.0.1.1.0-20598730"
         },
         "com.vmware.vsphere-ha": {
             "components": [
                 {
                     "component": "vsphere-fdm"
                 }
             ],
             "version": "8.0.2-22617221"
         }
     }
 }

Environment

VMware NSX 

VMware vSphere ESXi 8.x

Cause

• There is a discrepancy in vmw-nsx-lcp component installed on the ESXi Host (localcli_software-component-get.txt):

vmw-nsx-lcp_4.0.1.1.0.20598730-esx80:
   Name: nsx-lcp-bundle
   Display Name: NSX LCP Bundle
   Version: 4.0.1.1.0-8.0.20598730
   Display Version: NSX LCP Bundle(4.0.1.1.0-8.0.20598730)

and the vmw-nsx-lcp component being pushed by vLCM/NSX-MP (present under /repository/<MP version>/HostComponents/esx80):

vmw-nsx-lcp_4.0.1.1.0.20598730-esx80-snic-system:
   Name: nsx-lcp-bundle
   Display Name: NSX LCP Bundle
   Version: 4.0.1.1.0-8.0.20598730
   Display Version: NSX LCP Bundle(4.0.1.1.0-8.0.20598730)

• The discrepancy exists because the Hosts were manually prepared for NSX using an offline depot

Resolution

Re-prepare the ESXi Hosts/Cluster for NSX (delete and recreate TN which will push the correct nsx-lcp bundle to the host)