Virtual Machines experience guest OS crash in vSAN 7.X

Products

• VMware vSAN

Issue/Introduction

Symptoms:

Virtual machine running on different cluster was crashing almost around same time everyday. 

• Multiple VM from multiple cluster experiencing outage at the same time. 
• The PCPU utilization was going high. 
• There is a UNMAP spike in dom client ( 100ms) at that time.
• From vmware logs we found below event "The CPU has been disabled by the guest operating system" events are present in all the crash.

vsantraces: 

2025-03-22T00:04:25.200523 [279790114] [cpu52] [6ae6b298 OWNER unmap VMDISK] DOMTraceOperationNeedsRetry:4315: {'op': 0x45bd7b701680, 'objUuid': '75f0da67-2986-7984-XXXX-c470bd384034', 'pendingUpdatesListLen': 127, 'inclusiveCommitCount': 0, 'prepWaitQueueLen': 2523, 'status': 'VMK_LIMIT_EXCEEDED'}
2025-03-22T00:04:25.200562 [279790115] [cpu52] [6ae6b298 OWNER unmap VMDISK] DOMTraceOperationNeedsRetry:4315: {'op': 0x45bd7b827c80, 'objUuid': '75f0da67-2986-7984-XXXX-c470bd384034', 'pendingUpdatesListLen': 127, 'inclusiveCommitCount': 0, 'prepWaitQueueLen': 2524, 'status': 'VMK_LIMIT_EXCEEDED'}
2025-03-22T00:04:25.200600 [279790116] [cpu52] [6ae6b298 OWNER unmap VMDISK] DOMTraceOperationNeedsRetry:4315: {'op': 0x45bd7b0d5ac0, 'objUuid': '75f0da67-2986-7984-XXXX-c470bd384034', 'pendingUpdatesListLen': 127, 'inclusiveCommitCount': 0, 'prepWaitQueueLen': 2525, 'status': 'VMK_LIMIT_EXCEEDED'}

Environment

VMware vSAN 7.x

Cause

This issue may arise when large-sized unmaps are being split into smaller parts, causing them to occupy the 2PC queue because of the prepare limit. As a result, this can accumulate and lead to increased unmap latency.

Resolution

The recommendation is to move forward with the upgrade, as the issues are resolved in version 8.0 U3d

If it is for ESA, we strongly recommend to upgrade ESXi 8.0 P05 - 24674464 / ESXi 8.0 Update 3e due to critical fixes for data consistency and unmap.