ESXi v7.x intermittently disconnects from the vCenter
search cancel

ESXi v7.x intermittently disconnects from the vCenter

book

Article ID: 414146

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • The ESXi host periodically transitions to a "Not Responding" state in the vCenter inventory.
  • The hostd service on the ESXi host restarts unexpectedly and occasionally.
  • No core files are generated for hostd (found in /var/core/).
  • The /var/run/log/hostd-probe.log may not show any missed heartbeats, suggesting the service isn't hanging, but rather terminating or being killed.
  • The /var/run/log/hostd.log file is flooded with specific, repetitive entries (see the "Log Analysis" section below).

    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_spm_1.0.0, localId: spm
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_vmwarevmcrypt_1.0.0, localId: vmwarevmcrypt
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_vmwarelwd_1.0.0, localId: vmwarelwd
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] PluginLdr_Load: Loaded plugin 'libvmiof-disk-spm.so' from '/usr/lib64/vmware/plugin/libvmiof-disk-spm.so'
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] PluginLdr_Load: Loaded plugin 'libvmiof-disk-vmwarevmcrypt.so' from '/usr/lib64/vmware/plugin/libvmiof-disk-vmwarevmcrypt.so'
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] PluginLdr_Load: Loaded plugin 'libvmiof-disk-vmwarelwd.so' from '/usr/lib64/vmware/plugin/libvmiof-disk-vmwarelwd.so'
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] LWD: Initializing the LWD filter (process: 114416652)
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] LWD: Gathering filter's disk requirements
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] LWD: LwdFilter_DiskRequirements: allocation limits:
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] static memory: 34551896
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] memory per disk MB: 81
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] memory per IO: 10996
    YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=###ee3fc user=vpxuser] global memory: 33566568

Environment

  • VMware vSphere ESXi 7.0u3

Cause

The issue occurs mainly due to the glib library that is in use in VMware vSphere ESXi 7.0u3. An updated glib library is used in 8.0u2 onwards.

Resolution

Fixed in ESXi 8.0u2b.

Workaround for ESXi 7.0u3

  1. Upgrade/Patch to ESXi v7.0u3 P09
  2. Disable the vmwarelwd IOFilter.
    1. Create a python file, put the below lines into it and execute the script.

      #!/bin/python
      import vmware.vsi as vsi
      vsi.set('/iofilters/statusDisabledEvent', {'filterName': 'vmwarelwd', 'failureReason':'testing'})

    2. Check the IOFilter status through host MOB:

      • https://<ESXI_HOST_IP_ADDRESS>/mob/?moid=ha-host&doPath=config.ioFilterInfo

    3. The above should return/indicate that filter is not available, e.g.:

      NAME TYPE VALUE
      available boolean false
      id string "VMW_vmwarelwd_1.0.0"
      name string "vmwarelwd"
      releaseDate string "2021-07-21"
      summary string "LWD IO Filter"
      type string "dataCapture"
      vendor string "VMW"
      version string "1.0.0"

  3. Export hostd settings to the tmp.json file:

    configstorecli config current set -c esx -g services -k hostd > tmp.json

  4. Add a new property report_disabled_filters_in_host_config with the value of false to hostd settings object plugins/hostsvc. E.g.

      "plugins": {
    …
    .....
         "hostsvc": {
    …
    ......
     "report_disabled_filters_in_host_config": false,

  5. Set new hostd configuration from the tmp.json file, e.g.:

    configstorecli config current set -c esx -g services -k hostd -infile tmp.json

  6. Restart hostd service:

    /etc/init.d/hostd restart
  7. Wait for a minute/few for the hostd service to restart. To check that hostd has started successfully, run the following command to retrieve the list of VMs on the host:

    vim-cmd vmsvc/getallvms

Additional Information

Powered by Wolken Software