ESXi Cluster Image remediation fails with error "Remediation of Cluster Failed"

Products

• VMware vCenter Server

Issue/Introduction

- Although, Host upgrades successfully, with the Vendor Addons as well as the Firmware Components, the Host lands in Maintenance mode, and failing with the subsequent task from vLCM to exit the Maintenance mode, and move on to next host to remediate. 

- Cluster Level remediation fails with 'PostImage Update Task' fail Error as below;

 

/var/log/vmware/vmware-updatemgr/vum-server/vmware_vum_server.log

YYYY-MM-DDTHH:MM:SS.338Z error vmware-vum-server[4071305] [Originator@6876 sub=RemediateClusterTask opID=5xxxxx2-axxxb-4xxxx6-axxx3-3xxxxxxxxxx1] [Task, 524] Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:5xxxxx2-axxxb-4xxxx6-axxx3-3xxxxxxxxxx1. PostImage Firmware update failed on host esxi.host.fqdn.com(host-5041): Error:
 -->    com.vmware.vapi.std.errors.error
 --> Messages:
 -->    FROM_DOWNSTREAM: [vCenter: vcenter.fqdn.com][Cluster: CLUSTER_NAME][Host: esxi.host.fqdn.com][PostImage Update Task] The task with task id (task-1166) finished with errors.<>
 -->

Environment

ESXi 8.x

Dell OMEVV (iDRAC)

Cause

The error is from Dell OMEVV side: the POST_IMAGE_UPDATE task on HSM had progressed to 100 however the status was marked to "FAILED"

/var/log/vmware/vsm/hsm-service.log

YYYY-MM-DD HH:MM:SS,329[Dummy-35]hsmService:766 [INFO] Hsm service result: {'output': '{"action": "POST_IMAGE_UPDATE", "id": "task-1166", "description": "POST_IMAGE_UPDATE Task", 
"startTime": "YYYY-MM-DDTHH:MM:SS.006355Z", "estimatedTimeRemaining": 3600, "hosts": ["host-5041"], "status": "FAILED", "progress": 100, "operationStatusCode": 200, 
"messages": ["[vCenter: vcenter.fqdn.com][Cluster: CLUSTER_NAME][Host: esxi.host.fqdn.com][PostImage Update Task] 
The task with task id (task-1166) finished with errors."]}', 'error': None}

Resolution

The issue lies on the DELL OMEVV end, it is recommended to engage with DELL analyze and fix the issue.