vSphere HA restarts VM protected by vSphere Replication or similar 3rd party service

Products

• VMware vSphere ESXi

Issue/Introduction

• A VM crashes, has a core dump and is restarted by vSphere HA.
• Following log lines may be seen in the vmware.log file:

Cr(01) Upcall-25f8#### e662#### PANIC: VERIFY bora/esx/apps/dp/iofilter/lwdIoTracker.c:488
Wa(03) Upcall-25f8#### e662#### A core file is available in "/vmfs/volumes/633c3323-########-####-############/vm#############/vmx-zdump.000"
In(05) Upcall-25f8#### e662#### Backtrace:
Wa(03) mks - Panic in progress... ungrabbing
In(05) mks - MKS: Release starting (Panic)

• Following log lines may be observed in vmkernal.log file on the ESXi Host where the VM resides:

In(182) vmkernel: cpu39:209#####)uw.209##### (3893) requires 4 KB, asked 4 KB from iofiltervpd (641) which has 23552 KB occupied and 0 KB available.
In(182) vmkernel: cpu41:363#### opID=5c507f51)UserDump: 3072: vmx-Upcall-25f8:vm#############: Dumping cartel 363#### (from world 363####) to file /vmfs/volumes/633c3323-########-####-############/vm#############/vmx-zdump.000 ...
In(182) vmkernel: cpu49:363#### opID=5c50####)UserDump: 3367: vmx-Upcall-25f8:vm#############: Userworld(vmx-Upcall-25f8:vm#############) coredump complete.
In(182) vmkernel: cpu51:209####)Admission failure in path: host/iofilters/iofiltervpd:ioFilterVPServer.209####:uw.209####

Environment

• VMWare vSphere ESXi 7.x
• VMWare vSphere ESXi 8.x

Cause

• This happens due to there being more I/Os issued through the LWD I/O tracker than the maximum number of I/Os the VM is configured to handle.
• LWD (light weight delta) and SES (storage efficiency service) modules are leveraged by VM protection software such as vSphere Replication, Dell Recoverpoint etc.
• The I/O pattern for the VMDK of the VM is such that there are too many COW fragments that are written to the SES and the I/O tracker limit for number of tracked I/Os is under provisioned for the SES case. The current limit is 1536 but the limit should be increased to 8000.

Resolution

This is a known issue and is fixed in ESXi version 8.0u3b and newer.

VMware ESXi 8.0 Update 3b Release Notes