High CPU usage and cluster performance degradation in vSAN ESA 8.0U2 due to stuck Object resynchronization

Products

• VMware vSAN

Issue/Introduction

ESXi hosts in a vSAN ESA 8.0U2 cluster exhibit extremely high CPU usage (spikes near 100%).

The CPU contention may migrate between hosts (e.g., after a host reboot or object ownership change).

One or more vSAN objects are identified as "stuck" in a resynchronization state under Monitor > vSAN > Resyncing Objects.

Virtual Machine performance is severely impacted, VMs may also become unresponsive.

In /var/log/vmkernel.log of the affected ESXi host, repeated warnings similar to the following may be seen:

WARNING: VSAN: VsanIoctlCtrlNodeCommon:3157: [OBJECT_UUID]: RPC to DOM op readPolicy returned: Transient storage condition, suggest retry

 

Environment

vSAN ESA 8.0U2

Cause

In certain scenarios, layout of a vSAN Object in the Express Storage Architecture (ESA) may enter into transient inconsistency state. When this occurs, the Distributed Object Manager (DOM) may enter into a processing loop while attempting to reconcile unhealthy layout of an object. Resulting repeated readPolicy operations executed by DOM may lead to high CPU usage for the host owning the Object.

Resolution

Open a ticket with Broadcom Support when the above symptoms are observed.