Error: ESXi 9.x Upgrade Fails at Remediation Step

Products

• VMware Cloud Foundation

Issue/Introduction

• When performing an ESXi host upgrade (e.g., 9.0.1 to 9.0.2) within VCF Operations 9.x, the remediation task fails at the "Remediate Step." 
  
  
• The failed task on VCF Operations shows below error
  
• Error in /var/log/vmware/vmware-updatemgr/vum-server/hsm-service.log on the vCenter contains the following error

  HsmService:[Dummy-11]hsmService:275 [INFO] Using cert-based side-car endpoint with URL 'http://localhost:1080/external-cert/http1/#########/3512/rest/ovm/vlcm'
  HsmService:[Dummy-11]hsmService:310 [INFO] Successfully initiated executor
  HsmService:[Dummy-11]hsmService:472 [ERROR] get_host_inventory failed with InternalServerError
  HsmService:[Dummy-11]hsmService:552 [ERROR] ['Failed to start inventory for host-##']
  HsmService:[Dummy-11]hsmService:559 [INFO] Got http response status code: 500
  HsmService:[Dummy-11]hsmService:570 [ERROR] Error code: 500
  HsmService:[Dummy-11]hsmService:573 [ERROR] Transient error: None
  HsmService:[Dummy-11]hsmService:809 [INFO] Hsm service result: {'output': 'null', 'error': {'errorCode': 500, 'command': ('hosts', 'get'), 'input': '{"requestContext": null, "optArguments": {"host_id": "host-##", "package": "HPE Synergy Service Pack", "version": "SY-2026.04.01.902"}}', 'args': {'status': 500, 'message': ['Failed to start inventory for host-33'], 'isTransient': 'null'}}}

Environment

VMware Cloud Foundation 9.0.x

Cause

The failure occurs because Hardware compatibility validation failed when using the hardware vendor's Hardware Support Manager (HSM) plugin with VMware vSphere Lifecycle Manager (vLCM). When attempting to retrieve hardware firmware version information, the Hardware Support Manager (HSM) may return an internal error in response to the inventory request.

 

Resolution

• The issue falls outside of Broadcom's scope.
• Reach out to the hardware vendor responsible for providing and managing the firmware and driver updates accessed by vLCM through the HSM.
• HSMs are listed on the HCL https://compatibilityguide.broadcom.com/search?program=hsm&persona=live&column=partnerName&order=asc
• Prior to contacting the vendor, verify that the physical servers (and their management IP addresses) are fully registered, licensed, and actively monitored within the vendor's external management console (e.g., HPE OneView, Dell OME).
• For any additional information, reach out to Broadcom Support.