ESXi Upgrade Pre-Check Error: 'Error on Fetching Device Firmware Data, Device Compatibility Must Be Verified Manually'

Products

• VMware vSAN
• VMware vCenter Server
• VMware vSphere ESXi
• VMware SDDC Manager

Issue/Introduction

 

• Warning Message in the "Updates" Tab of Cluster related to Hardware Compatibility : “Error on fetching device firmware data, Device compatibility must be verified manually”

• The hardware compatibility for the storage controller indicates an error message 'Device is incompatible with the driver/firmware in cluster's image.' 
  
  

 

• Compliance results may be shown as unknown
  
  
  • In the /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log log of the vCenter Server, "Internal Server Error" are observed coming from HSM (Hardware Support Manager)

  [YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[520353] [Originator@6876 sub=Hsl::Util opID=#########-#######-#####-####-###########] [ErrorUtil 139] Internal Server Error
[YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[520353] [Originator@6876 sub=Hsl::ScanManager opID=#########-#######-#####-####-###########] [ScanManager 90] Failed to Initiate HW Scan task on hsm com.dell.plugin.OpenManager_HWSupportManager, Error Type : 8, Error Code : 500
[YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[520353] [Originator@6876 sub=Hsl::ScanManager opID=#########-#######-#####-####-###########] [ScanManager 162] Failed to trigger scan on hsm com.dell.plugin.OpenManager_HWSupportManager with Error:
-->    com.vmware.vapi.std.errors.internal_server_error
--> No messages!
--> scan results will be marked as unavailable
[YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[520353] [Originator@6876 sub=TelemetryError opID=#########-#######-#####-####-###########] [PManTelemetryEvents 85] Error messages are empty

  • When attempting to retrieve hardware firmware version information, the Hardware Support Manager (HSM) may return an internal error in response to the inventory request. As a result, it fails to retrieve the necessary hardware details, including firmware versions.

  [YYYY-MM-DDTHH:MM:SS] info vmware-vum-server[12468] [Originator@6876 sub=HCL::HostHclValidate] [HostHclValidate 760] Spec driver version match but Firmware NOT match in Device Constrants.Spec Firmware Version [ 24.15.03.00 ], Device Firmware Version [ 24.15.14.00 ]
[YYYY-MM-DDTHH:MM:SS] info vmware-vum-server[521299] [Originator@6876 sub=HCL::HostHclValidate] [HostHclValidate 760] Spec driver version match but Firmware NOT match in Device Constrants.Spec Firmware Version [ 17.15.08.00 ], Device Firmware Version [  ]

  • HSM logs will indicate the following log entries

  Path: /var/log/vmware/vmware-updatemgr/vum-server/hsm-service.log

  HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:801 [INFO] Running command from C++: hosts scan
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:754 [INFO] Running command hosts scan
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:248 [INFO] Got operation arguments: _OptArgsHolder(task_id=None, host_id=None, package=None, version=None, release=None)
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:258 [INFO] Initiating executor
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:483 [INFO] Got network location: <HSM IP Address>
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:508 [INFO] Extracted hostname: <HSM IP Address>, port: 443
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:274 [INFO] Successfully initiated executor
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:540 [ERROR] Unexpected error occurred during execution: Unable to get field messages, no field of that name found.
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:547 [INFO] Got http response status code: 500
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:558 [ERROR] Error code: 500
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:561 [ERROR] Transient error: None
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:797 [INFO] Hsm service result: {'output': 'null', 'error': {'errorCode': 500, 'command': ('hosts', 'scan'), 'input': '{"requestContext": null, "requestBody": {"action": "SCAN", "targetHosts": ....... }

  In an SDDC environment cluster remediation may pass the upgrade pre-check and initiate but then the upgrade task fails after successfully patching the first host in the cluster due to this issue.

  In the lifecycle manager logs on the sddc manger you see:

  remediation completed, compliance check reported host as 'non-compliant'. The image on the host does not match the image set for the cluster. Retry the cluster remediation operation.

Environment

• VMware vCenter Server 7.x 
  VMware vCenter Server 8.x 
• VMware vSphere ESXi 7.x 
  VMware vSphere ESXi 8.x 
• VMware vSAN 7.x
  VMware vSAN 8.x 
• SDDC manager 5x

Hardware Support Manager Integration such as

• HPE OneView for VMware vCenter 
• OpenManage Enterprise Integration for VMware vCenter (OMEVV) 
• Lenovo XClarity Integrator for VMware vCenter

Cause

Hardware compatibility validation failed when using the hardware vendor's Hardware Support Manager (HSM) plugin with VMware vSphere Lifecycle Manager (vLCM).

Note: The Hardware Support Manager (HSM) is a software solution provided by the hardware vendor, typically delivered as an OVF (Open Virtualization Format) image. Once deployed, the HSM includes a plug-in that registers itself as an extension within the vCenter Server. This integration allows for streamlined hardware management and support, enabling enhanced compatibility checks and firmware updates within the vSphere environment.

 

 

Resolution

• The issue falls outside of Broadcom's scope.
• Reach out to the hardware vendor responsible for providing and managing the firmware and driver updates accessed by vLCM through the HSM.
• HSMs are listed on the HCL https://compatibilityguide.broadcom.com/search?program=hsm&persona=live&column=partnerName&order=asc
• For any additional information, reach out to Broadcom Support. 

Additional Information

ESXi Upgrade via SDDC Manager blocked due to multiple Precheck Failures related to hardware firmware and compatibility checks.