Remediation task completes successfully without patching hosts in vSphere Lifecycle Manager (vLCM)

Products

VMware vCenter Server

Issue/Introduction

In vCenter, a remediation task completes without error, however the status for the baseline shows a status of "Missing".
The cluster's single image remediation process fails with the following error: "Upgrade failed during post-checks. Remediation reported success, but some hosts are not upgraded."
Parallel remediation is enabled in vCenter under Lifecycle Manager > Settings > Baselines > Edit Cluster Remediation Settings.
To validate the configuration, check the logs below:
- Refer to the SDDC Manager logs below: /var/log/vmware/vcf/lcm/lcm.log

YYYY-MM-DDTHH:MM:SS INFO [vcf_lcm,#############################,####] [c.v.e.s.l.s.i.ImageManagementServiceImpl,Scheduled-#] Domain version matrix with patch aware Dayn feature is : VersionMatrix(productVersions=[ProductVersion(productType=VCENTER, version=8.0.3.00800-25197330), ProductVersion(productType=NSX_T_MANAGER, version=4.2.3.3.0-25171318), ProductVersion(productType=ESX, version=8.0.3-25205845)])
YYYY-MM-DDTHH:MM:SS INFO [vcf_lcm,####################,####] [c.v.v.v.v.PersonalityImageManager,Precheck-#] Status of task :Info (com.vmware.cis.task.info) => {
progress = Progress (com.vmware.cis.task.progress) => {
total = 100,
completed = 100,
message = LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = com.vmware.vcIntegrity.lifecycle.Task.Progress,
defaultMessage = Current progress for task created by VMware vSphere Lifecycle Manager,
args = [],
params = <null>,
localized = Current progress for task created by VMware vSphere Lifecycle Manager
}
},
result = com.vmware.esx.settings.cluster_compliance => {incompatible_hosts=[host-######], hosts=[map-entry => {value=com.vmware.esx.settings.host_compliance => {image=<unset>, components=[], solutions=[], impact=UNKNOWN, commit=##, removed_components=<unset>, compliance_status_details=<unset>, add_on=com.vmware.esx.settings.add_on_compliance => {current=<unset>, stage_status=<unset>, notifications=com.vmware.esx.settings.notifications => {warnings=<unset>, errors=<unset>, info=<unset>}, status=UNAVAILABLE, target=<unset>}, stage_status=<unset>, hardware_support=<unset>, base_image=com.vmware.esx.settings.base_image_compliance => {current=com.vmware.esx.settings.base_image_info => {details=<unset>, version=}, stage_status=<unset>, notifications=com.vmware.esx.settings.notifications => {warnings=<unset>, errors=<unset>, info=<unset>}, status=UNAVAILABLE, target=com.vmware.esx.settings.base_image_info => {details=<unset>, version=}}, data_processing_units_compliance=<unset>, impact_details=<unset>, scan_time=YYYY-MM-DDTHH:MM:SS, remediation_details=<unset>, notifications=com.vmware.esx.settings.notifications => {warnings=<unset>, errors=[com.vmware.esx.settings.notification => {retriable=<unset>, id=com.vmware.vcIntegrity.lifecycle.HostScan.TaskApiError, originator=<unset>, time=YYYY-MM-DDTHH:MM:SS, message=com.vmware.vapi.std.localizable_message => {args=[NAME_OF_THE_ESXi, Error:
com.vmware.vapi.std.errors.not_allowed_in_current_state
Messages:
com.vmware.esx.task.contention<Another task is in progress. Please retry later.>
], default_message=A failure occurred when starting a host compliance check operation on host 'NAME_OF_THE_ESXi' : Error:
com.vmware.vapi.std.errors.not_allowed_in_current_state
Messages:
com.vmware.esx.task.contention<Another task is in progress. Please retry later.>
, localized=A failure occurred when starting a host compliance check operation on host 'NAME_OF_THE_ESXi' : Error:
com.vmware.vapi.std.errors.not_allowed_in_current_state
Messages:
com.vmware.esx.task.contention<Another task is in progress. Please retry later.>
, id=com.vmware.vcIntegrity.lifecycle.HostScan.TaskApiError, params=<unset>}, type=<unset>, resolution=<unset>}], info=<unset>}, status=UNAVAILABLE}, key=host-######}, map-entry => {value=com.vmware.esx.setti ... esx.settings.host_info => {is_vsan_witness=<unset>, name=NAME_OF_THE_ESXi}, key=host-######}, map-entry => {value=com.vmware.esx.settings.host_info => {is_vsan_witness=<unset>, name=NAME_OF_THE_ESXi}, key=host-######}], status=UNAVAILABLE},
description = LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = com.vmware.vcIntegrity.lifecycle.Task.Description,
defaultMessage = Task created by VMware vSphere Lifecycle Manager,
args = [],
params = <null>,
localized = Task created by VMware vSphere Lifecycle Manager
},
service = com.vmware.esx.settings.clusters.software,
operation = scan$task,
parent = ,
target = DynamicID (com.vmware.vapi.std.dynamic_ID) => {
type = ClusterComputeResource,
id = domain-#######
},
status = SUCCEEDED,
cancelable = true,
error = <null>,
startTime = YYYY-MM-DDTHH:MM:SS[GMT],
endTime = YYYY-MM-DDTHH:MM:SS[GMT],
user = [email protected]
[dynamic fields]: {
last_update_time = YYYY-MM-DDTHH:MM:SS,
subtasks = <unset>,
notifications = com.vmware.esx.settings.notifications => {warnings=[com.vmware.esx.settings.notification => {retriable=<unset>, id=com.vmware.vcIntegrity.lifecycle.HostScan.TaskApiErrorRetryWarning, originator=<unset>, time=YYYY-MM-DDTHH:MM:SS, message=com.vmware.vapi.std.localizable_message => {args=[NAME_OF_THE_ESXi], default_message=Retrying the host compliance check operation on host 'NAME_OF_THE_ESXi' because there is another operation in progress on this host and it can process one request at a time., localized=Retrying the host ...

- Refer to the vCenter logs below: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

YYYY-MM-DDTHH:MM:SS info vmware-vum-server[######] [Originator@#### sub=ScanUtil opID=########-####-####-####-############] [ScanUtil ###] Calling host NAME_OF_THE_ESXi (host-######) scan API
YYYY-MM-DDTHH:MM:SS error vmware-vum-server[863106] [Originator@#### sub=ScanUtil opID=########-####-####-####-############] [ScanUtil ###] Failed to scan host - (host = host-######, host name = NAME_OF_THE_ESXi) - (
error = com.vmware.vapi.std.errors.not_allowed_in_current_state) - (details = Error:
--> com.vmware.vapi.std.errors.not_allowed_in_current_state
--> Messages:
--> com.vmware.esx.task.contention<Another task is in progress. Please retry later.>
--> )
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[######] [Originator@6876 sub=InventoryTree opID=########-####-####-####-############] [InventoryTree ###] [ReleaseWriteLock] Released write lock on node host-######. Post unlock, notifie
d all the waiting readers and writers.
YYYY-MM-DDTHH:MM:SS warning vmware-vum-server[######] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask opID=########-####-####-####-############] [Task, ###] Task:com.vmware.vcIntegrity.lifecycle.Desir
edScanClusterTask ID:########-####-####-####-############. Calling host-###### host Image Scan request failed. Retriable error (= true) - Retry count = 11 - Error: Error:
--> com.vmware.vapi.std.errors.error
--> Messages:
--> com.vmware.vcIntegrity.lifecycle.HostScan.TaskApiError<A failure occurred when starting a host compliance check operation on host 'NAME_OF_THE_ESXi' : Error:
--> com.vmware.vapi.std.errors.not_allowed_in_current_state
--> Messages:
--> com.vmware.esx.task.contention<Another task is in progress. Please retry later.>
--> >
-->
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[######] [Originator@6876 sub=Telemetry opID=########-####-####-####-############] [TelemetryManager ###] New item added to telemetry data queue. Queue size: 1
...
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[######] [Originator@#### sub=com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask opID=########-####-####-####-############] [Task, ###] Task:com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask ID:########-####-####-####-############. Cluster image compliance result for cluster - #######-#####-##-### (domain-#######) : [
--> {
--> "STRUCTURE": {
--> "map-entry": {
--> "key": "host-######",
...
--> "STRUCTURE": {
--> "com.vmware.vapi.std.localizable_message": {
--> "args": [
--> "NAME_OF_THE_ESXi",
--> "Error:\n com.vmware.vapi.std.errors.not_allowed_in_current_state\nMessages:\n com.vmware.esx.task.contention<Another task is in progress. Please retry later.>\n"
--> ],
--> "default_message": "A failure occurred when starting a host compliance check operation on host 'NAME_OF_THE_ESXi' : Error:\n com.vmware.vapi.std.errors.not_allowed_in_current_state\nMessages:\n com.vmware.esx.task.contention<Another task is in progress. Please retry later.>\n",
--> "id": "com.vmware.vcIntegrity.lifecycle.HostScan.TaskApiError",
--> "localized": {
--> "OPTIONAL": null
--> },
--> "params": {
--> "OPTIONAL": null
--> }
--> }
--> }

YYYY-MM-DDTHH:MM:SS.###Z info vmware-vum-server [Originator@#### sub=Telemetry] [TelemetryManager ###] Sending telemetry data: {"@type":"pman_effective_coordinator_policy","taskId":"####|###","entityId":"####|domain-c####","failureAction":"RETRY","failureActionRetryDelay":300,"failureActionRetryCount":3,"preRemediationPowerAction":"DO_NOT_CHANGE_VMS_POWER_STATE","enableQuickBoot":true,"disableDpm":true,"disableHac":false,"evacuateOfflineVms":true,"enforceHclValidation":false,"parallelRemediationEnabled":true,"maxHostsForParallelRemediation":0,"enforceQuickPatch":false}

- RemediateClusterTask is created and starts running for cluster domain-c###.
  
  Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:###. Task Created
  Set com.vmware.vcIntegrity.lifecycle.RemediateClusterTask (###) progress to 1
  Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:###. Starting cluster(domain-c####) remediation with commitId(##)
- Cluster Compliance check finishes successfully with reporting hosts as NON_COMPLIANT.
  
  Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:###. Starting compliance scan for cluster domain-####(host_fqdn)
  Task:com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask ID:###. Compliance for host host-id
- After the scan completes, RemediateClusterTask attempts to fetch the host group but finds zero hosts for the cluster, logging the message: No hosts for cluster: domain-c#####

- No hosts are reported for Remediation.
  
  RemediateClusterTask - FinalizeTaskResult - remaining hosts set size : 0

Note: Cluster level remediation in vCenter also experiences this issue, where the task completes without applying patches to the hosts.

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
SDDC Manager 5.x

Cause

This issue occurs because parallelRemediationEnabled is set to enabled. With parallel remediation enabled, the process only applies to hosts that are already in Maintenance Mode, meaning vSphere lifecycle manager does not remediate any ESXi hosts that are not currently in maintenance mode.

Resolution

To resolve the patching failure when parallel remediation is enabled, implement one of the following options:

Option 1:

Place the host in maintenance mode manually before initiating the remediation.

Option 2:

Disable the parallel remediation option in vCenter.

Log in to vCenter Server via vSphere Client.
Navigate to Home > Lifecycle Manager.
Select Settings > Baselines (or Images for image-based clusters).
Click [EDIT].
Un-check Parallel remediation and click [SAVE].
Proceed with the remediation.

Note:

In image-based clusters (vLCM), this setting is located under Lifecycle Manager > Settings > Images > Edit Cluster Remediation Settings.
In some scenarios, parallel remediation is enabled at the cluster level. It can be disabled by navigating to: vSphere Client > Cluster > Updates > Hosts > Image > Image Compliance > Edit Cluster Remediation Settings.

Additional Information

Managing Host and Cluster Lifecycle