CLOMD stopped on more than 1 host in a VSAN cluster and crashes nearly immediately after restart

Products

• VMware vSAN

Issue/Introduction

Hosts cannot enter maintenance mode and clomd service is crashing with a backtrace

Environment

ESXi 7.0.3 EP13 24585291

Cause

In the logs you can see the following   

node must have exactly 1 delta component and 1 base component. Found 2 child components out of which  0 are delta components

And this is followed by clomd backtrace in clomd.log


[msg.log.error.unrecoverable] vSAN Cluster level Object Manager unrecoverable error: (host-xxxxxxx)

Resolution

Confirm that the flag is set correctly on all the hosts in the cluster 

vsish -e get /config/VSAN/intOpts/DeltaComponent  >>>> If DeltaComponent config option is disabled stale deltas cannot be cleaned up.

If the base and delta are both in an active state but the delta component doesn't have VSAN_COMPONENT_FLAGS_IS_DELTA set in flags the clomd service will crash.

The easiest way to fix this issue is to storage vMotion all the objects where the delta component doesn't have VSAN_COMPONENT_FLAGS_IS_DELTA set in flags.

If no other storage is available then we need to clone the objects or the entire VM.