vSAN objects may show inaccessible after a cluster power outage or network outage.

Products

• VMware vSAN

Issue/Introduction

• vSAN objects may show inaccessible after a cluster power outage or network outage.

• VMs would show inaccessible after a power outage or Network changes on vSAN cluster.

Environment

• VMware vSAN 8.x

• VMware vSAN 9.x

• vSAN NVMe attached to Tri-mode controller.

Cause

• The objects would fail to initialize upon a cluster level outage occur due to either power outage or major network activity on a vSAN cluster where the vSAN NVMe disks attached to the Tri-mode controller which is unsupported. Ref: vSAN support of NVMe devices behind tri-mode controllers

• The object initialization can be validated using the below log snippets.

clomd.log:
2025-11-23T04:31:12.828Z Er(27) clomd[2154851] [Originator@6876] CLOMBalanceRebalanceDiskV2: Failed to check c5731d69-4e4e-f2c3-2bd2-##### delta attribute: Not found
2025-11-23T04:31:12.828Z Cr(26) clomd[2154851] [Originator@6876] CLOMBalanceCheckDeltaComp: Failed to lookup composite object 6a181c69-ea66-58fd-556a-##### for component 6a181c69-1cc2-aaff-9ebf-####: Not found

2025-11-23T05:34:28.574Z Er(27) clomd[2159897] [Originator@6876] Parse error at token 848. Message: Unexpected value type. Expression up to error: ..."isResyncRunning" i1) ("expectedFullProvisionedBytes" l76665192448)) 72be2169-e853-f477-7532-##### (
2025-11-23T05:34:28.574Z Cr(26) clomd[2159897] [Originator@6876] clomdb-CdbUpdateCompositeEntry: Cannot unmarshal configuration for object u:acf20b69-dafa-ec2c-fbe3-####:t:13:o:00000000-0000-0000-0000-000000000000(Hlth:Unhealthy:16):r:5(F4734F51A0) valueLength=8216: Bad parameter.
2025-11-23T05:34:28.574Z Cr(26) clomd[2159897] [Originator@6876] clomdb-Cdb_RetrieveNonSubscribedEntry: Failed to construct CdbObject: Bad parameter.

2025-11-23T05:34:28.574Z Er(27) clomd[2159897] [Originator@6876] Parse error at token 845. Message: Unexpected value type. Expression up to error: ..."componentState" l10) ("componentStateTS" l1763819353) ("staleLsn" (
2025-11-23T05:34:28.574Z Er(27) clomd[2159897] [Originator@6876] Parse error at token 845. Expected Token type: TokenListEnd. Message: Unexpected token type. Expression up to error: ..."componentState" l10) ("componentStateTS" l1763819353) ("staleLsn" (

• When trying to list the objects using 'esxcli vsan debug object list' the below events would show up in esxcli.log.

esxcli.log:

2025-11-22T17:52:16Z Er(11) esxcli[2145585]: list failed: No root object set for output. The command did not provide proper output.
2025-11-22T17:52:16Z Er(11)[+] esxcli[2145585]: Implementation error! The output of this command does not match the declared output type, see errors above.

• When checking for traces by support, vsantraces on hosts show below events:

2025-11-22T13:22:18.787443 [486660723] [cpu2] [CLIENT] DOMTraceObjectCleanupObjectTree:6364: {'objUuid': 'e5c90069-f23f-f3c6-d804-1096c646e83c', 'reason': 'server release', 'cleanupMode': 'DOM_OBJ_CLEANUP_UNKNOWN', 'releaseCleanupStarted': False, 'lvl 1': 0x42001b450be6, 'lvl 2': 0x42001ac1aca9}
2025-11-22T13:22:19.290555 [486661647] [cpu25] [CLIENT] DOMTraceObjectCleanupObjectTree:6364: {'objUuid': 'a4d21469-6efd-5108-b8ef-b4cadd1be1fc', 'reason': 'server release', 'cleanupMode': 'DOM_OBJ_CLEANUP_UNKNOWN', 'releaseCleanupStarted': False, 'lvl 1': 0x42001b450be6, 'lvl 2': 0x42001ac1aca9}

2025-11-23T05:33:55.598963 [75209696] [cpu3] [6ec6ea93 OWNER] DOMTraceObjectInitRootObjectComplete:887: {'objUuid': 'bb731d69-3014-cd6b-0772-####', 'obj': 0x45bb4ec873c0, 'status': 'VMK_BAD_PARAM', 'refCount': 1}
2025-11-23T05:33:55.598963 [75209697] [cpu3] [6ec6ea93 OWNER ROOT_OBJ_FACTORY] DOMTraceRootObjFactoryInitObj:11881: {'newObjUuid': 'bb731d69-3014-cd6b-0772-####', 'newAssoc': 0x0, 'newObj': 0x45bb4ec873c0, 'returnStatus': 'VMK_BAD_PARAM', 'opStatus': 'VMK_OK', 'jumpFrom': 11829}
2025-11-23T05:33:55.598963 [75209698] [cpu3] [6ec6ea93 OWNER ROOT_OBJ_FACTORY] DOMTraceRootObjFactoryDispatchCompleted:12330: {'obj': 0x45bb4ec873c0, 'newObjUuid': 'bb731d69-3014-cd6b-0772-####', 'factoryOp': 0x45bb074c5c40, 'status': 'VMK_BAD_PARAM', 'factoryOpState': 'INIT_OBJ', 'newObjType': 'OWNER'}
2025-11-23T05:33:55.598977 [75209702] [cpu3] [6ec6ea93] DOMTraceDeleteOwnerOnFailedElection:28195: {'objUuid': 'bb731d69-3014-cd6b-0772-####', 'setupState': 'OBJINIT', 'refCount': 2, 'retry': False, 'status': 'VMK_BAD_PARAM', 'ctx': 0x4323ffcff4a0}
2025-11-23T05:33:55.603648 [75209825] [cpu3] [6ec6ea93] DOMTraceOwnerInstantiateCompleted:28227: {'objUuid': 'bb731d69-3014-cd6b-0772-####', 'setupState': 'OWNER_ELECTION', 'refCount': 1, 'status': 'VMK_BAD_PARAM', 'ctx': 0x4323ffcff4a0}
2025-11-23T05:33:55.603648 [75209826] [cpu3] [6ec6ea93 COMP] DOMTraceComponentOwnerInitCb:6402: {'objUuid': 'bb731d69-3014-cd6b-0772-####', 'status': 'VMK_BAD_PARAM'}

Resolution

• In order to address the inaccessible VMs, you may have to restore from backup to a supported storage environment.

• NVMe devices are only supported directly connected to a PCIe slot on the bus. Please refer the article for more information on vSAN support of NVMe devices behind tri-mode controllers