vSAN Skyline Object Health intermittently drops to 60% and returns to normal within the next second.

Products

• VMware vSAN

Issue/Introduction

vSAN Skyline Health reports that the Object Health Summary is dropping regularly, indicating intermittent fluctuations in object health status.

Environment

VMware vSAN 8.#

Cause

The issue is caused by a transient ABSENT state in components, lasting less than half a second. This behaviour results from a race condition during component server movement within the DOM (Distributed Object Manager), leading to intermittent anomalies in the health report.

vSAN-traces shows: 

2025-06-26T04:10:30.447Z In(166) vsansystem[2100066]: [vSAN@6876 sub=Libs opId=56ec7699-54a3] Object 6e#####-####-####-####-00######### is not compliant, reason: 0x5, incomplainceFlags = 32. 

2025-06-26T04:10:30.047794 [220313831] [cpu53] [] DOMTraceConfigStatus:11050: {'objUuid': '6e#####-####-####-####-00#########', 'hasAvailability': False, 'hasQuorum': False, 'perfCompliant': True, 'hasUnhealthy': True, 'isStale': False, 'isAPD': False, 'isPDL': False, 'state': 12, 'deleteState': 0x0, 'checksumPolicy': 0x0, 'updateStatus': 'VMK_OK', 'configStateRevision-16': 0xffff, 'nextPossibleConfigStateRevision-16': 0x4902, 'CSN': 16614, 'hasEDA': False, 'compressionState': 0x0, 'zdomObject': False, 'SCSN': 16617}. 

Become healthy: 

2025-06-26T04:10:30.362450 [220325993] [cpu50] [] DOMTraceConfigStatus:11050: {'objUuid': '6e#####-####-####-####-00#########', 'hasAvailability': True, 'hasQuorum': True, 'perfCompliant': True, 'hasUnhealthy': False, 'isStale': False, 'isAPD': False, 'isPDL': False, 'state': 7, 'deleteState': 0x0, 'checksumPolicy': 0x0, 'updateStatus': 'VMK_OK', 'configStateRevision-16': 0x4909, 'nextPossibleConfigStateRevision-16': 0xffff, 'CSN': 16625, 'hasEDA': False, 'compressionState': 0x0, 'zdomObject': False, 'SCSN': 16624}. 

 

DOMTraceConfigStatus: 

2025-06-26T04:10:30.039244 [220313637] [cpu53] [LEAF] DOMTraceLeafUpdateConnectionState:2418: {'obj': 0x45db0f863d00, 'objUuid': '6e#####-####-####-####-00#########', 'oldFlags': 0x0, 'newFlags': 0x0, 'objState': 'absent', 'connType': 'RDT', 'newConnState': 'CONNECTED', 'oldConnState': 'DISCONNECTED', 'newAggrConnState': 'CONNECTED', 'oldAggrConnState': 'DISCONNECTED'}

Connected back: 

2025-06-26T04:10:30.266676 [220323199] [cpu50] [e6f68883 LEAF establishRDT] DOMTraceObjectEstablishRDTResult:2343: {'op': 0x45dafd7702c0, 'objUuid': '6e#####-####-####-####-00#########', 'status': 'VMK_OK', 'assoc': 0x432382b4e1c0}
2025-06-26T04:10:30.266677 [220323200] [cpu50] [LEAF] DOMTraceLeafUpdateConnectionState:2418: {'obj': 0x45db0f863d00, 'objUuid': '6e#####-####-####-####-00#########', 'oldFlags': 0x1000080, 'newFlags': 0x1000080, 'objState': 'absent', 'connType': 
'RDT', 'newConnState': 'CONNECTED', 'oldConnState': 'DISCONNECTED', 'newAggrConnState': 'CONNECTED', 'oldAggrConnState': 'DISCONNECTED'}

 

Resolution

This is a known issue. It occurs due to a brief, sub-second ABSENT state of vSAN components caused by a race condition during component server transitions in the Distributed Object Manager (DOM).
Although this does not impact production workloads, it results in intermittent drops in vSAN Skyline Health reports.

The issue is scheduled to be addressed in the upcoming patch release 8u3p07.