Host is unavailable for checking compliance.

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

When checking host profile compliance in vCenter Server, you experience these symptoms:

vCenter Server reports the error,

Host <hostname> is not in compliance with the attached profile: Datacenter <Name>, Host <Hostname>, Compute Resource <Cluster Name> : Quick stats on <hostname> in <Cluster> in <Datacenter> is not up-to-date"
The task times out in vCenter Server.
You see the error:

Host is unavailable for checking compliance
Attempting to apply a host profile times out after 5 minutes with the error:

The request failed because the remote server "vCenterFQDN" took too long to respond” Error Stack: Call "HostProfileManager.GenerateConfigTaskList" for object "HostProfileManager" on vCenter Server "vCenterFQDN" failed.

You have a host with more than 80 datastores presented or there is a storage device reporting a Permanent Device Loss(PDL). For more information, Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere ESXI
In the /var/log/hostd.log file, you see these errors:

The task begins:

<YYYY-MM-DD>T<time>Z [3CFC5B90 info 'TaskManager' opID=93992203-00005F6B-da-d4] Task Created : haTask--vim.profile.host.profileEngine.ComplianceManager.checkHostCompliance-511155668
</time>

VMFS Refresh called:

<YYYY-MM-DD>T<time></time>Z [3FE80B90 verbose 'Hostsvc::DatastoreSystem'] ReconcileVMFSDatastores called: refresh = true, rescan = false <YYYY-MM-DD>T<time></time>Z [3FE80B90 verbose 'FSVolumeProvider'] RefreshVMFSVolumes called

VMFS Refresh Finishes:

<YYYY-MM-DD>T<time></time>Z [3FE80B90 verbose 'Hostsvc::DatastoreSystem'] ReconcileVMFSDatastores: Done discovering new filesystem volumes.

A new datastore refresh is called:

<YYYY-MM-DD>T<time></time>Z [3C0DBB90 verbose 'FSVolumeProvider'] RefreshVMFSVolumes called

This pattern continues until the host profile compliance check times out after 10 minutes:

<YYYY-MM-DD>T<time></time>Z [3CAF0B90 error 'SOAP command stub adapter /usr/bin/sh' opID=93992203-00005F6B-da-d4] Error: timeout waiting for command <YYYY-MM-DD>T<time></time>Z [3CAF0B90 error 'Default' opID=93992203-00005F6B-da-d4] AdapterServer caught exception: Operation timed out

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server

VMware vSphere ESXi

Cause

The RefreshVMFSVolumes task can take a long time to complete if there are too many datastores or if there are underlying storage issues. For eg. a storage device reporting permanent device loss(PDL)

Resolution

To resolve the issue, identify why the RefreshVMFSVolumes task takes a long time to complete on the esxi host.

If a storage device is reporting a permanent device loss, please review the instructions in the kb article, Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere ESXI

To work around this issue, increase the vCenter Server and ESXi timeout values.

To adjust the Host Profile application timeout value

Log in to vCenter Server using the vSphere Client.
Navigate to Administration > vCenter Server Settings.
Click vCenter Server Settings.
Click Timeout Settings.
Change the values for Normal Operations and Long Operations to 7200.
Click OK to apply the changes.

To adjust the Host Profile compliance timeout values

Log in to your ESXi/ESX host with root privileges using the console or an SSH session.
Back up the cmdMo.xml file.

Note: By default this file is located at /etc/vmware/hostd.
Open the cmdMo.xml file using a text editor.
Locate this entry:

<moId> ha-hostprofileengine-compliancemanager </moId>
Change <timeOutInSeconds> 1800 </timeOutInSeconds> after the preceding entry:

For Example:

<moId> ha-hostprofileengine-compliancemanager </moId>
<timeOutInSeconds> 1800 </timeOutInSeconds>

For more information on increasing the vCenter Server and ESXi timeout values, see Extending the default timeout periods for vCenter Server, ESX/ESXi hosts, and vCenter Update Manager (1017253).