ESXi v7.x intermittently disconnects from the vCenter
search cancel

ESXi v7.x intermittently disconnects from the vCenter

book

Article ID: 414146

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

ESXi intermittently goes into "Not responding" state in the vCenter.

 

On the ESXi, hostd restarts occasionally. No hostd core files are generated. There might be no indication of failed hostd heartbeats in hostd-probe.log

hostd.log continuously records the following entries:

YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_spm_1.0.0, localId: spm
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_vmwarevmcrypt_1.0.0, localId: vmwarevmcrypt
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_vmwarelwd_1.0.0, localId: vmwarelwd
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] PluginLdr_Load: Loaded plugin 'libvmiof-disk-spm.so' from '/usr/lib64/vmware/plugin/libvmiof-disk-spm.so'
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] PluginLdr_Load: Loaded plugin 'libvmiof-disk-vmwarevmcrypt.so' from '/usr/lib64/vmware/plugin/libvmiof-disk-vmwarevmcrypt.so'
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] PluginLdr_Load: Loaded plugin 'libvmiof-disk-vmwarelwd.so' from '/usr/lib64/vmware/plugin/libvmiof-disk-vmwarelwd.so'
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] LWD: Initializing the LWD filter (process: 114416652)
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] LWD: Gathering filter's disk requirements
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] LWD: LwdFilter_DiskRequirements: allocation limits:
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] static memory: 34551896
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] memory per disk MB: 81
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] memory per IO: 10996
YYY-MM-DDTHH:MM:SS.ZZZZ info hostd[NNNNNNNNN] [Originator@6876 sub=Libs opID=XXXee3fc user=vpxuser] global memory: 33566568

 

Environment

ESXi v7.0u3

Cause

The issue occurs mainly due to the glib library that is in use in ESXi v7.0u3. An updated glib library is used in 8.0u2 onwards.

Resolution

Fixed in ESXi 8.0u2b.

 

Workaround for v7.0u3

  1. Upgrade/Patch to ESXi v7.0u3 P09
  2. Disable the vmwarelwd IOFilter.
    1. Create a python file, put the below lines into it and execute the script.
      1. #!/bin/python
        import vmware.vsi as vsi
        vsi.set('/iofilters/statusDisabledEvent', {'filterName': 'vmwarelwd', 'failureReason':'testing'})
    2. Check the IOFilter status through host MOB:
      1. https://<ESXI_HOST_IP_ADDRESS>/mob/?moid=ha-host&doPath=config.ioFilterInfo
    3. The above should return/indicate that filter is not available, e.g.:
      1. NAME TYPE VALUE
        available boolean false
        id string "VMW_vmwarelwd_1.0.0"
        name string "vmwarelwd"
        releaseDate string "2021-07-21"
        summary string "LWD IO Filter"
        type string "dataCapture"
        vendor string "VMW"
        version string "1.0.0"
  3. Export hostd settings to the tmp.json file:
    1. $ configstorecli config current set -c esx -g services -k hostd > tmp.json
  4. Add a new property report_disabled_filters_in_host_config with the value of false to hostd settings object plugins/hostsvc. e.g.


    1.    "plugins": {
      ...

      .....

           "hostsvc": {
      ...

      ......

       "report_disabled_filters_in_host_config": false,
       
  5. Set new hostd configuration from the tmp.json file, e.g.:
    1. $ configstorecli config current set -c esx -g services -k hostd -infile tmp.json
  6. Restart hostd service:
    1. $ /etc/init.d/hostd restart
       
  7. Wait for a minute/few for the hostd service to restart. To check that hostd has started successfully, run the following command to retrieve the list of VMs on the host:
    1. $ vim-cmd vmsvc/getallvms

Additional Information

Powered by Wolken Software