vSAN Cluster APD events and multiple virtual machines experienced HA event

Products

• VMware vSAN

Issue/Introduction

vCenter UI events notified APD event for VMs.

Virtual machines hosted in vSAN cluster also experienced HA events. 

 

Environment

vSAN 7.0X

vSAN 8.0x

Cause

ESXi host log events found at time of outage. The following log messages present symptomatic messages of a upstream physical networking issue. These messages indicate that the backing network for the vSAN vmkernel port traffic is latent and when the backing vSAN networking is unreliable (flapping) this behavior will that caused the vSAN datastore to become unreliable. 

In other words - A unreliable network will cause VMs to loose access to their storage in vSAN and APD messages can be seen in the vCenter UI events.

 

vmkwarning.log

2025-10-21T0X:38:54.104Z Wa(180) vmkwarning: cpu84:2101797)WARNING: RDT: RDTSetMessageTxState:5153: CMMDS RDTTraceSlowMessageTx addr 192.168.3.12 delay (4195) oldTxState 4 newTxState 7

- RDTTraceSlowMessageTx: means slow networking.

 

vobd.log

2025-10-21T02:38:59.458Z In(14) vobd[2098028]  [vmfsCorrelator] 12644810932637us: [vob.vmfs.heartbeat.timedout] XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

- vob.vmfs.heartbeat.timedout: means headbeats for VM are timing out. Once a timeout threshold is met, VMs will HA.

 

 

vmkernel.log

2025-10-21T02:38:54.109Z In(182) vmkernel: cpu84:2102341)CMMDS: CMMDSUtil_PrintArenaEntry:98: XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX: [542089617]:Adding a new Membership entry (XXXXXXXXXX-XXXXXXXXXXX) with 13 members:

2025-10-21T02:39:00.123Z In(182) vmkernel: cpu94:2102341)CMMDS: CMMDSUtil_PrintArenaEntry:98: XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX: [542091508]:Adding a new Membership entry (XXXXXXXXXX-XXXXXXXXXXX) with 6 members:

2025-10-21T02:39:06.388Z In(182) vmkernel: cpu87:2102341)CMMDS: CMMDSUtil_PrintArenaEntry:98: XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX: [542100374]:Adding a new Membership entry (XXXXXXXXXX-XXXXXXXXXXX) with 2 members

-  Adding a new Membership entry: means that expected vSAN ESXi hosts are reconnecting to the cluster.

Resolution

Investigate upstream physical network for service data reliability and performance issues.

Additional Information

If further assistance is needed to identify this type of issue please open a support case with Broadcom Support.