VMs goes unresponsive due to high UNMAP burst

Products

• VMware vSAN

Issue/Introduction

• Database virtual machines experience severe performance degradation, high guest OS I/O wait times, or complete database unresponsiveness despite minimal application traffic.

• The vCenter Server and hostd service log a loss of the VMware Tools heartbeat as the guest OS enters an unresponsive state:

• At the hypervisor layer, the VM remains powered on, but guest-level I/O stalls completely due to storage queue saturation.
  
  In vmware.log below events 
026-08-22T23:03:20.052Z In(05) vcpu-35 - TOOLS: appName=toolbox, oldStatus=1, status=0, guestInitiated=1.
2026-08-22T23:03:20.121Z In(05) vmx - GuestRpc: Got RPCI vsocket connection 6, assigned to channel 0.
2026-08-22T23:03:20.122Z In(05) vmx - Guest: [vgauthservice] VGAuthService END SERVICE
2026-08-22T23:03:20.122Z In(05) vmx - GuestRpc: Closing channel 0 connection 6
2026-08-22T23:03:42.000Z In(05) vcpu-0 - Tools: Tools heartbeat timeout.
2026-08-22T23:03:42.001Z In(05) vcpu-0 - Tools: Running status rpc handler: 1 => 0.
2026-08-22T23:03:42.001Z In(05) vcpu-0 - Tools: Changing running status: 1 => 0.

Environment

vSAN ESA 8.x

Cause

When a guest OS issues an abnormally high, sustained volume of concurrent UNMAP (TRIM/discard) commands targeting a vSAN ESA object, the requests can saturate all 64 available PVSCSI queue slots.

To protect regular read/write I/O from resource contention, vSAN ESA automatically throttles UNMAP requests. However, under heavy concurrent discard bursts, this throttling behavior leads to the following sequence:

1. Queue Saturation: Heavy UNMAP commands fill all 64 PVSCSI queue slots.

2. I/O Blocking: Because all available queue slots are occupied by throttled UNMAP operations, standard read and write requests are blocked from dispatching.

3. Guest OS Stall: While underlying vSAN storage-layer write latencies remain low, the guest OS experiences a complete I/O stall, causing VMware Tools heartbeats to time out and database services to freeze.

Resolution

Workaround

To prevent aggressive UNMAP throttling from blocking PVSCSI queue slots, adjust the zDOM UNMAP timeout parameters on the ESXi host. Reducing these parameters restricts UNMAPs to a maximum delay of 1ms, releasing PVSCSI queue slots rapidly for standard I/O

esxcfg-advcfg -s 1 /VSAN/zDOMUnmapTimeoutMsec
esxcfg-advcfg -s 1 /VSAN/zDOMUnmapMaxTimeoutMsec

• VMware Engineering is working on the long term fix.