ESXi host was not responding as "Too many open files" with hostd

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

ESXi host is unresponsive/ non-responding in vCenter.

In the hostd log, messages related to "Too many open files" in relation to VM flat files.

--/var/run/log/hostd.log

YYYY-MM-DDTHH:MM:SS In(166) Hostd[2100321]: [Originator@6876 sub=DiskLib opID=####-##-#### sid=#### user=vpxuser:user01] DISKLIB-VMFS  : "/vmfs/volumes/########/VM_name/VM_name-flat.vmdk" :failed to open (Reached the host's limit for open files): Directory open failed on '/vmfs/volumes/########/VM_name': Too many open files (24). Type 3
YYYY-MM-DDTHH:MM:SS Er(163) Hostd[2100312]: [Originator@6876 sub=SoapAdapter.HTTPService] Failed to accept connection; <server <acceptor p:0x000000#######, h:51, <TCP '127.0.0.1 : 8307'>>, >, N7Vmacore15SystemExceptionE(Too many open files)

Environment

VMware ESXi (All Versions)

Cause

Check the current amount of vmdk files opened by hostd.

# lsof | grep hostd | grep VM_name-flat.vmdk | wc -l

 550

The hostd.log, shows the same vpxuser account for all the open tasks 

Resolution

Trace the source of the open files task from the vpxuser account 

   Then stop these tasks in the vCenter or backup software.

As a workaround, restart the below host management agents to restore access to the host in vCenter.

# /etc/init.d/hostd restart

# /etc/init.d/vpxa restart

Additional Information

Restarting Management Agents in ESXi