Host profile compliance checks time out in environments with a large number of datastores and stateless cache enabled
search cancel

Host profile compliance checks time out in environments with a large number of datastores and stateless cache enabled

book

Article ID: 343987

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

When checking host profile compliance in vCenter Server, you experience these symptoms:

  • The task times out and stateless caching is enabled.
  • You see the error:

    Host is unavailable for checking compliance

  • Attempting to apply a host profile times out after 5 minutes with the error:

    The request failed because the remote server "vCenterFQDN" took too long to respond” Error Stack: Call "HostProfileManager.GenerateConfigTaskList" for object "HostProfileManager" on vCenter Server "vCenterFQDN" failed.

    You have a host with more than 80 datastores presented.

  • In the /var/log/hostd.log file, you see these errors:
The task begins:

<YYYY-MM-DD>T<time>Z [3CFC5B90 info 'TaskManager' opID=93992203-00005F6B-da-d4] Task Created : haTask--vim.profile.host.profileEngine.ComplianceManager.checkHostCompliance-511155668
</time>
VMFS Refresh called:

<YYYY-MM-DD>T<time></time>Z [3FE80B90 verbose 'Hostsvc::DatastoreSystem'] ReconcileVMFSDatastores called: refresh = true, rescan = false <YYYY-MM-DD>T<time></time>Z [3FE80B90 verbose 'FSVolumeProvider'] RefreshVMFSVolumes called

VMFS Refresh Finishes:

<YYYY-MM-DD>T<time></time>Z [3FE80B90 verbose 'Hostsvc::DatastoreSystem'] ReconcileVMFSDatastores: Done discovering new filesystem volumes.

A new datastore refresh is called:

<YYYY-MM-DD>T<time></time>Z [3C0DBB90 verbose 'FSVolumeProvider'] RefreshVMFSVolumes called

This pattern continues until the host profile compliance check times out after 10 minutes:

<YYYY-MM-DD>T<time></time>Z [3CAF0B90 error 'SOAP command stub adapter /usr/bin/sh' opID=93992203-00005F6B-da-d4] Error: timeout waiting for command <YYYY-MM-DD>T<time></time>Z [3CAF0B90 error 'Default' opID=93992203-00005F6B-da-d4] AdapterServer caught exception: Operation timed out

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.



Environment

VMware vCenter Server 5.1.x
VMware vCenter Server 5.5.x
VMware vCenter Server 5.0.x
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.0

Cause


Resolution

This is a known issue affecting ESXi 5.0, 5.1, and 5.5.
This issue is resolved in these ESXi versions:
To work around this issue, increase the vCenter Server and ESXi timeout values.

To adjust the Host Profile application timeout value

  1. Log in to vCenter Server using the vSphere Client.
  2. Navigate to Administration > vCenter Server Settings.
  3. Click vCenter Server Settings.
  4. Click Timeout Settings.
  5. Change the values for Normal Operations and Long Operations to 7200.
  6. Click OK to apply the changes.

To adjust the Host Profile compliance timeout values

Note: The hostd timeouts configured through the cmdMo.xml are increased in ESXi 5.1 Patch 7 and ESXi 5.5 Update 3. This is independent of the speed increases provided in the other releases.

If you are running ESXi 5.0, ensure to make these changes:
  1. Log in to your ESXi/ESX host with root privileges using the console or an SSH session. For more information, see:
  1. Back up the cmdMo.xml file.

    Note: By default this file is located at /etc/vmware/hostd.
  2. Open the cmdMo.xml file using a text editor.
  3. Locate this entry:

    <vim.profile.host.profileEngine.HostProfileManager> and </vim.profile.host.profileEngine.HostProfileManager>

  4. Change to:

    <vim.profile.host.profileEngine.HostProfileManager>
    <timeOutInSeconds> 1800 </timeOutInSeconds>
    </vim.profile.host.profileEngine.HostProfileManager>
Note: VMware recommends you to revert the timeout changes after upgrading to one of the builds that resolves this issue.
If you are running ESXi 5.1, ensure to make these changes:
  1. Log in to your ESXi/ESX host with root privileges using the console or an SSH session. For more information, see:
  1. Back up the cmdMo.xml file.

    Note: By default this file is located at /etc/vmware/hostd.

  2. Open the cmdMo.xml file using a text editor.
  3. Locate this entry:

    <moId> ha-hostprofileengine-compliancemanager </moId>

  4. Add <timeOutInSeconds> 1800 </timeOutInSeconds> after the preceding entry:

    For Example:

    <moId> ha-hostprofileengine-compliancemanager </moId>
    <timeOutInSeconds> 1800 </timeOutInSeconds>

    For more information on increasing the vCenter Server and ESXi timeout values, see Extending the default timeout periods for vCenter Server, ESX/ESXi hosts, and vCenter Update Manager (1017253).
An alternative workaround is to disable stateless caching and upgrade the hosts.
To verify if stateless caching is enabled:
  1. Log in to vCenter Server using the vSphere Client.
  2. Navigate to Home > Management > Host Profiles.
  3. Right-click the host profile and click Edit Profile.
  4. Expand System Image Cache Configuration.


Additional Information

Tech Support Mode for Emergency Support
Unable to connect to an ESX host using Secure Shell (SSH)
Correlating build numbers and versions of VMware products
Extending the default timeout periods for vCenter Server, ESX/ESXi hosts, and vCenter Update Manager
VMware ESXi 5.0, Patch ESXi500-201405401-BG: Updates esx-base
VMware ESXi 5.1, Patch ESXi-5.1.0-20140704001-standard
VMware ESXi 5.5, Patch ESXi550-Update02: ESXi 5.5 Complete Update 2
多数のデータストアがある環境でステートレス キャッシュが有効化されていると、ホスト プロファイルのコンプライアンス チェックがタイムアウトする
在启用了大量数据存储和无状态缓存的环境中检查主机配置文件合规性超时

Impact/Risks:


Powered by Wolken Software