vLCM compliance check tasks intermittently stall at various progress stages.

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter environment was upgraded from version 7.x to 8.0 U3x.
• The cluster is configured with a defined image that includes several additional components.
• To maintain compliance and ensure a smooth upgrade, all components must be up to date and aligned with the image baseline.
• Outdated components can cause inconsistencies, leading to compliance deviations or potential upgrade failures.
• A performance assessment using vimtop indicated that the Update Manager's CPU utilization exceeds 100%.
  
  vimtop

1776184 updatem+  20   0 4830.0m   2.1g   0.0   7.2   0:00.00 S          `- /u+
1776189 updatem+  20   0 4830.0m   2.1g  95.3   7.2  27:51.46 R          `- /u+
1776926 updatem+  20   0   11.7g 298.5m   0.0   1.0   0:00.00 S          `- jr+
1776937 updatem+  20   0   11.7g 298.5m   0.0   1.0   0:02.17 S              `+

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

 

YYYY-MM-DDTHH:MM:SS info vmware-vum-server[1776189] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.CreateRecommendationTask] [Task 627] Set com.vmware.vcIntegrity.lifecycle.CreateRecommendationTask () progress to 65
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[1815272] [Originator@6876 sub=DraftsManager] [DraftsManager 672] Successfully retrieved desired Spec for the entity: domain-c##
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[1815272] [Originator@6876 sub=ScanUtil] [ScanUtil 1156] Successfully retrieved software spec for the entity: <cluster_name> (domain-c##)
-->  {
-->     "add_on": null,
-->     "alternative_images": null,
-->     "base_image": {
-->         "version": "7.0.3-0.150.24784741"
-->     },
-->     "components": {
-->         "Broadcom-ELX-lpfc": "14.0.543.0-1OEM.700.1.0.15843807",
-->         "Broadcom-bnxt-Net-RoCE": "220.0.165.0-1OEM.700.1.0.15843807",
-->         "Broadcom-lsi-mr3": "7.720.04.00-1OEM.700.1.0.15843807",
-->         "Broadcom-lsi-msgpt35": "20.00.00.00-1OEM.700.1.0.15843807",
-->         "Intel-Volume-Mgmt-Device": "2.7.2.1001-1OEM.700.1.0.15843807",
-->         "Intel-i40en": "2.2.7.0-1OEM.700.1.0.15843807",
-->         "Intel-icen": "1.7.5.0-1OEM.702.0.0.17630552",
-->         "Intel-igbn": "1.8.2.0-1OEM.700.1.0.15843807",
-->         "Intel-irdman": "1.3.6.0-1OEM.700.1.0.15843807",
-->         "Intel-ixgben": "1.12.2.0-1OEM.700.1.0.15843807",
-->         "MRVL-E3-Ethernet-iSCSI-FCoE": "3.0.202.0-1OEM.700.1.0.15843807",
-->         "MRVL-E4-CNA-Driver-Bundle": "5.0.339.0-1OEM.700.1.0.15843807",
-->         "MRVL-QLogic-FC": "5.1.68.0-1OEM.703.0.0.18644231",
-->         "Mellanox-nmlx5": "4.21.71.101-1OEM.702.0.0.17630552",
-->         "Microchip-smartpqi": "70.4252.0.103-1OEM.700.1.0.15843807",
-->         "Pensando-ionic-en": "18.0.0-1OEM.700.1.0.15843807"
-->     },
-->     "hardware_support": null,
-->     "removed_components": null,
-->     "solutions": {
-->         "com.vmware.vsphere-ha": {
-->             "components": [
-->                 {
-->                     "component": "vsphere-fdm"
-->                 }
-->             ],
-->             "version": "8.0.3-24853646"
-->         }
-->     }
--> }

 

YYYY-MM-DDTHH:MM:SS info vmware-vum-server[1815272] [Originator@6876 sub=Downloader] [Downloader 76] DownloadVibsForSpec called.
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[1815268] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.CreateRecommendationTask] [Task, 524] Task:com.vmware.vcIntegrity.lifecycle.CreateRecommendationTask ID:########-####-####-####-############. Task Created


Environment

VMware vCenter Server 8.x
VMware vSphere ESXi 7.x

Cause

The issue seems to originate from components that were manually added to the cluster image outside of the standard configuration. Because these components do not match the versions defined in the intended cluster image, a discrepancy has developed between the actual host setup and the baseline image.
This misalignment prevents the environment from maintaining compliance and requires manual remediation. To resolve it, the manually added components must either be updated to the correct version or downgraded, ensuring they are consistent with the cluster’s defined image.

Resolution

To fully resolve the issue and bring the environment into alignment, the ESXi hosts should be upgraded to version 8.0, which provides the necessary updates and compatibility improvements.

Workaround:
As a potential workaround, there are two approaches you can consider:

1) Switching from image-based to baseline-based cluster management – This allows more flexibility in handling components since the environment will no longer be strictly tied to a single predefined image. However, it may reduce consistency across hosts.
2) Creating and applying a custom image – A tailored image can be built to include all required components specific to your environment, ensuring both compliance and consistency across the cluster.