ESXi "hostdstats" Partition at 100% Usage Causing "Not Responding" or "Disconnected" Status in vSphere Inventory

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Host is disconnected from vCenter Server
• Cannot interact with host directly through the vSphere Client
• The hostAgentStats-20.stats file unexpectedly grows large
• The output of the vdf -h command shows that hostdstats has 100% space used:
  

  Ramdisk Size Used Available Use% Mounted on
  MAINSYS 32M 21M 10M 68% --
  tmp 192M 60K 191M 0% --
  updatestg 750M 24K 749M 0% --
  hostdstats 166M 166M 0B 100% --
  

• On ESXi, the /var/log/hostd.log file may contain an entry similar to:
  
  

  Error: "FileIO error: No space left: /var/lib/vmware/hostd/stats/hostAgentStats.idMap"
  
  

• On ESXi, the /var/log/hostd.log file may contain an entry similar to:
  
  

  HostCtl Exception during stats collection: Error interacting with configuration file /etc/vmware/esx.conf: Failed attempting to lock file. Another process has locked the file for more than 20 seconds. The process holding the lock is hostd (#########). This operation will complete if it is run again after the lock is released. 

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x 

Cause

The host is unable to write performance information to the hostAgentStats-20.stats file due to the hostdstats partition being full.

Resolution

1. Backup hostAgentStats-20.stats with the command:
   • cp /var/lib/vmware/hostd/stats/hostAgentStats-20.stats /var/tmp

2. Remove the hostAgentStats-20.stats with the command:
   1. rm /var/lib/vmware/hostd/stats/hostAgentStats-20.stats
3. Restart the host's Management agents. For more information, see Restarting the Management agents in ESXi. This will cause the host to automatically recreate the hostAgentStats-20.stats file.
4. (Applicable only if Host status is "DISCONNECTED"): Right-click the host and reconnect it to vCenter Server.

Additional Information