Host is unavailable for checking compliance.

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

When checking host profile compliance in vCenter Server, these symptoms may be experienced:

• vCenter Server reports the error,
  
  Host <hostname> is not in compliance with the attached profile: Datacenter <Name>, Host <Hostname>, Compute Resource <Cluster Name> : Quick stats on <hostname> in <Cluster> in <Datacenter> is not up-to-date"
  
  
• The task times out in vCenter Server.
• The error is seen:
  
  Host is unavailable for checking compliance
  
• Attempting to apply a host profile times out after 5 minutes with the error:
  
  The request failed because the remote server "vCenterFQDN" took too long to respond” Error Stack: Call "HostProfileManager.GenerateConfigTaskList" for object "HostProfileManager" on vCenter Server "vCenterFQDN" failed.
  
  A host with more than 80 datastores presented or there is a storage device reporting a Permanent Device Loss(PDL).  For more information, Permanent Device Loss (PDL) and All-Paths-Down (APD) on host 
• In the /var/log/hostd.log file, these errors are seen:

The task begins:

<YYYY-MM-DD>T<time>Z [3CFC5B90 info 'TaskManager' opID=93992203-00005F6B-da-d4] Task Created : haTask--vim.profile.host.profileEngine.ComplianceManager.checkHostCompliance-511155668
</time>

 

VMFS Refresh called:

<YYYY-MM-DD>T<time></time>Z [3FE80B90 verbose 'Hostsvc::DatastoreSystem'] ReconcileVMFSDatastores called: refresh = true, rescan = false <YYYY-MM-DD>T<time></time>Z [3FE80B90 verbose 'FSVolumeProvider'] RefreshVMFSVolumes called

 

VMFS Refresh Finishes:

<YYYY-MM-DD>T<time></time>Z [3FE80B90 verbose 'Hostsvc::DatastoreSystem'] ReconcileVMFSDatastores: Done discovering new filesystem volumes.

 

A new datastore refresh is called:

<YYYY-MM-DD>T<time></time>Z [3C0DBB90 verbose 'FSVolumeProvider'] RefreshVMFSVolumes called

 

This pattern continues until the host profile compliance check times out after 10 minutes:

<YYYY-MM-DD>T<time></time>Z [3CAF0B90 error 'SOAP command stub adapter /usr/bin/sh' opID=93992203-00005F6B-da-d4] Error: timeout waiting for command <YYYY-MM-DD>T<time></time>Z [3CAF0B90 error 'Default' opID=93992203-00005F6B-da-d4] AdapterServer caught exception: Operation timed out

Environment

• VMware vCenter Server
• VMware vSphere ESXi

Cause

The RefreshVMFSVolumes task can take a long time to complete if there are too many datastores or if there are underlying storage issues. For e.g. a storage device reporting permanent device loss(PDL)

Resolution

To resolve the issue, identify why the RefreshVMFSVolumes task takes a long time to complete on the esxi host.

 

If a storage device is reporting a permanent device loss, please review the instructions in the kb article, Permanent Device Loss (PDL) and All-Paths-Down (APD) on host

 

To work around this issue, increase the vCenter Server and ESXi timeout values.

To adjust the Host Profile application timeout value

1. Log in to vCenter Server using the vSphere Client.
2. Navigate to Administration > vCenter Server Settings.
3. Click vCenter Server Settings.
4. Click Timeout Settings.
5. Change the values for Normal Operations and Long Operations to 7200.
6. Click OK to apply the changes.

To adjust the Host Profile compliance timeout values

1. Log in to the ESXi/ESX host with root privileges using the console or an SSH session.
2. Back up the cmdMo.xml file.
   
   Note: By default this file is located at /etc/vmware/hostd.
   
   
3. Open the cmdMo.xml file using a text editor.
4. Locate this entry:
   
   <moId> ha-hostprofileengine-compliancemanager </moId>
   
   
5. Change <timeOutInSeconds> 1800 </timeOutInSeconds> after the preceding entry:
   
   For Example:
   
   <moId> ha-hostprofileengine-compliancemanager </moId>
<timeOutInSeconds> 1800 </timeOutInSeconds>

Additional Information