vSphere Cluster remediation fails with error "compliance results for the host <hostname> are unavailable"

Products

• VMware vSphere ESXi
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

vLCM Image based vSphere Cluster remediation attempt fails for all the hosts in the cluster with the error  "Compliance results for the host <hostname> are unavailable"

When the check compliance is run, it will return unknown status once it is completed.

The image includes the Firmware and Driver Addons

Cluster Update page has a the banner "Has some hardware compatibility issue. Refer to vCenter server logs for further details"

Environment

VMware vSphere 7.x

VMware vSphere 8.x 

Hardware Support Manager integration (such as DELL OpenManage, HPE OneView, and Lenovo XClarity)

Cause

This issue can occur when the vLCM is unable to communicate to the HSM to obtain the current firmware of the host.

Lack of firmware data will cause the vLCM to complete it's compliance check which will result in an unknown state of the compliance check.

 

vmware-vum-server.log will have entries similar to the below

YYYY-MM-DDThh:mm:ss.zzzZ error vmware-vum-server[1386488] [Originator@6876 sub=Hsl::Util opID=xxxxxx-yyyy-aaaaa-bbbb-123abc123abc] [ErrorUtil 139] Internal Server Error
YYYY-MM-DDThh:mm:ss.zzzZ error vmware-vum-server[1386488] [Originator@6876 sub=Hsl::ScanManager opID=xxxxxx-yyyy-aaaaa-bbbb-123abc123abc] [ScanManager 90] Failed to Initiate HW Scan task on hsm com.hpe.com4vc.vlcmhsm, Error Type :          8, Error Code : 500
YYYY-MM-DDThh:mm:ss.zzzZ error vmware-vum-server[1386488] [Originator@6876 sub=Hsl::ScanManager opID=xxxxxx-yyyy-aaaaa-bbbb-123abc123abc] [ScanManager 162] Failed to trigger scan on hsm com.hpe.com4vc.vlcmhsm with Error:
-->    com.vmware.vapi.std.errors.internal_server_error
--> No messages!
--> scan results will be marked as unavailable


YYYY-MM-DDThh:mm:ss.zzzZ  error vmware-vum-server[1386492] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask opID=xxxxxx-yyyy-aaaaa-bbbb-123abc123abc] [Task, 524] Task:com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask ID:aaaaaaa-zzzz-dddd-bbbb-567def567def. Failed to invoke HW scan on one or more HSM. Error :%1
YYYY-MM-DDThh:mm:ss.zzzZ  warning vmware-vum-server[1386492] [Originator@6876 sub=Hsl::Util::ScanUtil opID=xxxxxx-yyyy-aaaaa-bbbb-123abc123abc] [ScanUtil 181] Failed to convert hsmHostComplianceMap to             hostHsmHspComplianceMap and hostHsmRemediationImpactMap


 

hsm-service.log

 

HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-13]hsmService:436 [ERROR] scan_host failed with InternalServerError
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-13]hsmService:540 [ERROR] ['Failed to obtain SPP details:Gen11 Service Pack for ProLiant from HPE GreenLake for Compute Ops Management.']
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-13]hsmService:547 [INFO] Got http response status code: 500
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-13]hsmService:558 [ERROR] Error code: 500
HsmService:YYYY-MM-DD hh:mm:ss,zzz[Dummy-13]hsmService:561 [ERROR] Transient error: None

 

In this log excerpts, the events related to HPE HSM is shown. It may vary a bit for the other hardware.

Resolution

Review the HSM appliance status for possible connectivity issues and potential certificate related issues.

Re-integrate the HSM plugin with the vCenter with the help of the hardware vendors.

Additional Information

https://www.vmware.com/docs/introducing-vsphere-lifecycle-management-vlcm

The warning "Has some hardware compatibility issue. Refer to vCenter server logs for further details" can be displayed for various compatibility checks.

Make sure to review further about the hardware compatibility using the KB Checking VMware ESXi Host Compatibility Before Upgrade Using the Broadcom Compatibility Guide