ESXi Upgrade Pre-Check Error: 'Error on Fetching Device Firmware Data, Device Compatibility Must Be Verified Manually'
search cancel

ESXi Upgrade Pre-Check Error: 'Error on Fetching Device Firmware Data, Device Compatibility Must Be Verified Manually'

book

Article ID: 388422

calendar_today

Updated On:

Products

VMware vSAN VMware vCenter Server VMware vSphere ESXi VMware SDDC Manager / VCF Installer

Issue/Introduction

 

  • Warning Message in the "Updates" Tab of Cluster related to Hardware Compatibility : “Error on fetching device firmware data, Device compatibility must be verified manually”

  • The hardware compatibility for the storage controller indicates an error message 'Device is incompatible with the driver/firmware in cluster's image.' 

 

  • Compliance results may be shown as unknown

    • In the /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log log of the vCenter Server, "Internal Server Error" are observed coming from HSM (Hardware Support Manager)

    [YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[520353] [Originator@6876 sub=Hsl::Util opID=#########-#######-#####-####-###########] [ErrorUtil 139] Internal Server Error
    [YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[520353] [Originator@6876 sub=Hsl::ScanManager opID=#########-#######-#####-####-###########] [ScanManager 90] Failed to Initiate HW Scan task on hsm com.dell.plugin.OpenManager_HWSupportManager, Error Type : 8, Error Code : 500
    [YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[520353] [Originator@6876 sub=Hsl::ScanManager opID=#########-#######-#####-####-###########] [ScanManager 162] Failed to trigger scan on hsm com.dell.plugin.OpenManager_HWSupportManager with Error:
    -->    com.vmware.vapi.std.errors.internal_server_error
    --> No messages!
    --> scan results will be marked as unavailable
    [YYYY-MM-DDTHH:MM:SS] error vmware-vum-server[520353] [Originator@6876 sub=TelemetryError opID=#########-#######-#####-####-###########] [PManTelemetryEvents 85] Error messages are empty

    • When attempting to retrieve hardware firmware version information, the Hardware Support Manager (HSM) may return an internal error in response to the inventory request. As a result, it fails to retrieve the necessary hardware details, including firmware versions.

    [YYYY-MM-DDTHH:MM:SS] info vmware-vum-server[12468] [Originator@6876 sub=HCL::HostHclValidate] [HostHclValidate 760] Spec driver version match but Firmware NOT match in Device Constrants.Spec Firmware Version [ 24.15.03.00 ], Device Firmware Version [ 24.15.14.00 ]
    [YYYY-MM-DDTHH:MM:SS] info vmware-vum-server[521299] [Originator@6876 sub=HCL::HostHclValidate] [HostHclValidate 760] Spec driver version match but Firmware NOT match in Device Constrants.Spec Firmware Version [ 17.15.08.00 ], Device Firmware Version [  ]

    • HSM logs will indicate the following log entries

    Path: /var/log/vmware/vmware-updatemgr/vum-server/hsm-service.log

    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:801 [INFO] Running command from C++: hosts scan
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:754 [INFO] Running command hosts scan
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:248 [INFO] Got operation arguments: _OptArgsHolder(task_id=None, host_id=None, package=None, version=None, release=None)
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:258 [INFO] Initiating executor
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:483 [INFO] Got network location: <HSM IP Address>
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:508 [INFO] Extracted hostname: <HSM IP Address>, port: 443
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:274 [INFO] Successfully initiated executor
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:540 [ERROR] Unexpected error occurred during execution: Unable to get field messages, no field of that name found.
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:547 [INFO] Got http response status code: 500
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:558 [ERROR] Error code: 500
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:561 [ERROR] Transient error: None
    HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-7]hsmService:797 [INFO] Hsm service result: {'output': 'null', 'error': {'errorCode': 500, 'command': ('hosts', 'scan'), 'input': '{"requestContext": null, "requestBody": {"action": "SCAN", "targetHosts": ....... }

    hsmService.log may also contain explicit failure reasons during the SCAN action, such as: {"messages": ["Error: Compliance results unavailable for host host-XXXX. Reason: Unable to find host in HPE OV."]}"
    You may also  observe the GET_HOST_INVENTORY action returning a 500 Internal Server Error.

    In an SDDC environment cluster remediation may pass the upgrade pre-check and initiate but then the upgrade task fails after successfully patching the first host in the cluster due to this issue.

    In the lifecycle manager logs on the sddc manger you see:

    remediation completed, compliance check reported host as 'non-compliant'. The image on the host does not match the image set for the cluster. Retry the cluster remediation operation.

Environment

  • VMware vCenter Server 7.x 
    VMware vCenter Server 8.x 
  • VMware vSphere ESXi 7.x 
    VMware vSphere ESXi 8.x 
  • VMware vSAN 7.x
    VMware vSAN 8.x 
  • SDDC manager 5x

Hardware Support Manager Integration such as

  • HPE OneView for VMware vCenter 
  • OpenManage Enterprise Integration for VMware vCenter (OMEVV) 
  • Lenovo XClarity Integrator for VMware vCenter

Cause

Hardware compatibility validation failed when using the hardware vendor's Hardware Support Manager (HSM) plugin with VMware vSphere Lifecycle Manager (vLCM). When attempting to retrieve hardware firmware version information, the Hardware Support Manager (HSM) may return an internal error in response to the inventory request.

This typically occurs when:

  • The physical ESXi hosts have not been successfully registered or added to the external vendor hardware appliance's inventory (e.g., the nodes are missing from the HPE OneView database).
  • The HSM plugin service experiences a processing crash when parsing the inventory data fetched from the baseboard management controllers (iLO/iDRAC/XCC).

Note: The Hardware Support Manager (HSM) is a software solution provided by the hardware vendor, typically delivered as an OVF (Open Virtualization Format) image. Once deployed, the HSM includes a plug-in that registers itself as an extension within the vCenter Server. This integration allows for streamlined hardware management and support, enabling enhanced compatibility checks and firmware updates within the vSphere environment.

 

 

Resolution

  • The issue falls outside of Broadcom's scope.
  • Reach out to the hardware vendor responsible for providing and managing the firmware and driver updates accessed by vLCM through the HSM.
  • HSMs are listed on the HCL https://compatibilityguide.broadcom.com/search?program=hsm&persona=live&column=partnerName&order=asc
  • Prior to contacting the vendor, verify that the physical servers (and their management IP addresses) are fully registered, licensed, and actively monitored within the vendor's external management console (e.g., HPE OneView, Dell OME).
  • For any additional information, reach out to Broadcom Support. 

Additional Information

ESXi Upgrade via SDDC Manager blocked due to multiple Precheck Failures related to hardware firmware and compatibility checks.

Powered by Wolken Software