ESXi host hostd service crashes periodically during VM backup job

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi host periodically shows as "Disconnected" or "Not Responding" in vCenter Server.
• 3rd party backup operations (e.g., Commvault, NetBackup, Veeam) fail or disrupt during the disconnect.
• The hostd process crashes and generates core dumps.
• vmkernel logs indicate Out of Memory (OOM) for hostd: 

  uw.#### (####) requires 1024 KB, asked 1024 KB from hostd (722) which has #### KB occupied and 0 KB available.

• hostd logs show large block list entries for the Blklistsvc service: 

  Built block list with #### entries. Failed to open virtual disk: [] / Reached the host's limit for open files 40.

Environment

• VMware vSphere ESXi 7.x, 8.x, 9.x
• 3rd party backup software using VADP/CBT API

Cause

This issue occurs when a backup job triggers a request for an extremely large block list (CBT). This typically happens when the target Virtual Machine has a large snapshot chain or the backup job includes a very high volume of data or VMs simultaneously. The Blklistsvc service in hostd exhausts available memory or file descriptors while building the list, leading to a service crash.

Resolution

To restore immediate connectivity:

1. Restart management agents on the affected ESXi host: 

   /etc/init.d/hostd restart
   /etc/init.d/vpxa restart

To prevent future crashes:

1. Commit snapshots: Ensure the Virtual Machine does not have a large or long-standing snapshot chain before the backup starts.
2. Optimize backup jobs: Split large backup jobs into smaller groups and stagger the schedules to reduce concurrent resource demands on the host.