hostd core dumps due to file descriptor exhaustion on an ESXi host running VMs with NVIDIA vGPU pass-through configuration

Products

• VMware vSphere ESXi

Issue/Introduction

On an ESXi host running multiple VMs with NVIDIA GPUs passed through in vGPU mode, the hostd process core dumps and terminates abnormally.

Immediately before the hostd crash, logs similar to the following may be output to hostd.log:

Er(163) Hostd[2099490]: [Originator@6876 sub=Libs] error [ConfigStore:41b3d40700] Failed to connect to sqlite /etc/vmware/schemastore/schema-store-1(unable to open database file) rc= 14 (current connections=1)
Er(163) Hostd[2099478]: [Originator@6876 sub=Libs] error [ConfigStore:41b2b87700] [2000] Failed to connect to database

Environment

VMware ESXi 8.0 Update 3

 

Cause

The hostd process, in conjunction with NVIDIA components, opens file descriptors (FDs) for vmgfx character devices and uses FDs up to the upper limit.
When FDs are exhausted, hostd fails to connect to the internal SQLite database (/etc/vmware/schemastore/schema-store-1), generates a core dump, and restarts.

Resolution

Resolution:

This issue is currently under investigation by Broadcom Engineering. To receive updates on this issue, Subscribe to this knowledge article to get updates on this issue.

Workaround:

• Checking for signs of FD leak progression:
  You can check the progression of the hostd vmgfxX FD leak by running the following command on a running ESXi host, even if impacts (such as virtual machine power-on failures) have not yet manifested.

  vmkvsitools lsof | grep hostd-worker | grep vmgfx | wc -l

   

• Action before impacts manifest (Preventative Measure):
  If the number of open vmgfxX devices exceeds 1,000, this issue can be prevented by proactively restarting the hostd service.
  
  Reference: KB 320280: Restarting Management Agents in ESXi

• Action if impacts have manifested:
  If impacts such as virtual machine power-on failures persist even after restarting the hostd service, evacuate the virtual machines from the ESXi host, place the host into maintenance mode, and reboot it to recover the NVIDIA components.

 

Additional Information

Problem determination from core dump:

By running the following command against the generated hostd core dump (zdump), you can confirm the FD leak of the vmgfx device file. If the output number reaches 4000, it may indicate that the host is affected by this issue.

Command execution example:

vmkdump_extract -f /var/core/hostd-zdump.000 |grep -c vmgfx
3960

 

Japanese KB: NVIDIA GPU を vGPU モードでパススルーした VM を多数稼働させている ESXi ホストにおいて、hostd プロセスがコアダンプし異常終了する（442819）