vSAN Witness Appliance fails to join cluster with Out of Memory (OOM) and CMMDS Admission Failure
search cancel

vSAN Witness Appliance fails to join cluster with Out of Memory (OOM) and CMMDS Admission Failure

book

Article ID: 438199

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

A vSAN Witness Appliance fails to enable clustering or join a vSAN cluster. Management services on the Witness host are unresponsive or failing to initialize.

  • vSAN clustering and directory services fail to enable.

  • The following error appears in vmkernel.log: WARNING: CMMDS: CMMDSEnable:2675: Failed to enable vSAN clustering and directory services: Out of memory.

  • Admission failures are observed for system heaps: Admission failure in path: host:opt:vsan:cmmds-90:heap:CMMDSUpdates... WARNING: Heap: 1081: Could not create a pagepool for heap CMMDSUpdates... Admission check failed for memory resource

From the Vmkernel.log of the witness appliance were 52c9a971-fcda-821e-ceac-###########  is the UUID of the cluster you are adding to this witness. 

26-04-24T18:49:46.716Z In(182) vmkernel: cpu1:321017 opID=8e9c1967)Admission failure in path: host:opt:vsan:cmmds-90:heap:CMMDSUpdates5b98c67b+0x1d3
2026-04-24T18:49:46.716Z In(182) vmkernel: cpu1:321017 opID=8e9c1967)heap:CMMDSUpdates5b98c67b+0x1d3 (256724) requires 69804 KB, asked 69804 KB from host (0) which has 16643784 KB occupied and 28940 KB available.
2026-04-24T18:49:46.716Z Wa(180) vmkwarning: cpu1:321017 opID=8e9c1967)WARNING: Heap: 1081: Could not create a pagepool for heap CMMDSUpdates5b98c67b: Admission check failed for memory resource
2026-04-24T18:49:46.716Z Wa(180) vmkwarning: cpu1:321017 opID=8e9c1967)WARNING: CMMDS: CMMDSHeapCreate:2631: 52c9a971-fcda-821e-ceac-###########: CMMDS heap creation failed: CMMDSUpdates, min 71471699, max 71471699
2026-04-24T18:49:46.716Z Wa(180) vmkwarning: cpu1:321017 opID=8e9c1967)WARNING: CMMDS: CMMDSSetupContext:2403: 52c9a971-fcda-821e-ceac-###########: Failed to initialize update memory: Out of memory
2026-04-24T18:49:46.716Z In(182) vmkernel: cpu1:321017 opID=8e9c1967)CMMDS: CMMDSDeinitUpdateMem:617: 52c9a971-fcda-821e-ceac-###########: Max from preallocated 0, Max from heap = 0, sizeof Update = 488
2026-04-24T18:49:46.716Z In(182) vmkernel: cpu1:321017 opID=8e9c1967)CMMDS: CMMDSArenaVMKRemoveClusterOffsetsEntry:2591: 52c9a971-fcda-821e-ceac-###########: Removed offsets entry from cluster tree for 52c9a971-fcda-821e-ceac-###########
2026-04-24T18:49:46.716Z In(182) vmkernel: cpu1:321017 opID=8e9c1967)CMMDS: CMMDSArenaVMKDeinitialize:1015: 52c9a971-fcda-821e-ceac-###########: BlockSize 256, maxBlocksUsed 23, memoryAtMaxBlocks 5172, maxMemoryAllocated 5172
2026-04-24T18:49:46.718Z In(182) vmkernel: cpu1:321017 opID=8e9c1967)Destroyed VSAN Slab cmmds-rdtHdrSlab (maxCount=0 failCount=0)
2026-04-24T18:49:46.718Z In(182) vmkernel: cpu1:321017 opID=8e9c1967)Destroyed VSAN Slab cmmds-rdtSGSlab (maxCount=0 failCount=0)
2026-04-24T18:49:46.721Z Wa(180) vmkwarning: cpu1:321017 opID=8e9c1967)WARNING: CMMDS: CMMDSEnable:2675: Failed to enable vSAN clustering and directory services: Out of memory.

Environment

VMware ESXi (All Versions)

Cause

The Witness Appliance is undersized for the amount of 2-node clusters assigned to it. The "Tiny" deployment profile (16 GB RAM) cannot accommodate the memory heap requirements (CMMDS) needed to track the current volume of vSAN objects and metadata.

  • Tiny (16 GB RAM): Supports up to 10 nodes and 1,000 components. 

  • Medium (32 GB RAM): Supports up to 500 nodes and 21,000 components. 

  • Large (64 GB RAM): Supports over 500 nodes and 45,000 components.

Resolution

Workaround

  • Identify Current Sizing: Confirm what profile was used to deoply the Witness appliance by checking the amount of Memory allocated to the witness VM.

  • Resize the Appliance:

    • Power off the Witness Appliance VM.

    • Increase the RAM to 32 GB (Medium Profile) or 64 GB (Large Profile) based on the cluster size.

    • Set a 100% Memory Reservation for the Witness VM to ensure physical RAM is always available to the VMkernel.

  • Reboot: Power on the Witness Appliance and verify that CMMDSEnable completes successfully in the vmkernel.log

Fix 

  • Deploy a new witness appliance sized correctly for your environment 
Powered by Wolken Software