vSAN skyline health show clomd service status as "ABNORMAL" and the service crash when manually started on the affected ESXi host.

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• vSAN skyline health check show "CLOMD" service status as "Abnormal" on the ESXi host.

• Attempting to manually start the service results in it stopping or crashing after some time.

• The clomd logs show the following messages. Prior to the crashing, "ObjFmtChg in progress" messages are noticed along with the workItem type as PROACTIVE_REBALANCE. However, when the resync status is checked for the proactive rebalance, there is no resync running.

  2025-07-23T07:27:23.868Z info clomd[164559335] [Originator@6876 opID=#########] CLOMReconfigure: Reconfiguring ########-####-####-#####-############ workItem type PROACTIVE_REBALANCE
  2025-07-23T07:27:23.878Z info clomd[164559335] [Originator@6876 opID=#########] CLOMFixObjFmtChgInProgress: ObjFmtChg in progress. Fixing transient tree first
  2025-07-23T07:27:23.879Z PANIC: NOT_REACHED ####/###/####/vsan_config_builder.c:744
  2025-07-23T07:27:23.879Z Backtrace:
  2025-07-23T07:27:23.880Z Backtrace[14] 0000030efb18ac40 rip=00000059af36fd5d rbx=0000000000000000 rbp=0000000000000000 r12=000000596d6fe844 r13=0000030efb18ad08 r14=0000000000000000 r15=0000000000000000
  2025-07-23T07:27:23.880Z Backtrace[15] 0000030efb18ad00 rip=000000596d6fe86d rbx=0000000000000000 rbp=0000000000000000 r12=000000596d6fe844 r13=0000030efb18ad08 r14=0000000000000000 r15=0000000000000000
  2025-07-23T07:27:23.880Z Backtrace[16] 0000030efb18ad08 rip=0000000000000000 rbx=0000000000000000 rbp=0000000000000000 r12=000000596d6fe844 r13=0000030efb18ad08 r14=0000000000000000 r15=0000000000000000
  2025-07-23T07:27:23.880Z SymBacktrace[0] 0000030efb188660 rip=00000059ae85dd7f in function Panic_Panic in object /lib64/libvmlibs.so loaded at 00000059ae6b7000
  2025-07-23T07:27:23.880Z SymBacktrace[1] 0000030efb188aa0 rip=00000059ae85de4b in function Panic in object /lib64/libvmlibs.so loaded at 00000059ae6b7000
  2025-07-23T07:27:23.880Z SymBacktrace[2] 0000030efb188b80 rip=000000596d80027b in function (null) in object /usr/lib/vmware/vsan/bin/clomd loaded at 000000596d6e4000
  2025-07-23T07:27:23.880Z SymBacktrace[10] 0000030efb18aa10 rip=000000596d77caf3 in function (null) in object /usr/lib/vmware/vsan/bin/clomd loaded at 000000596d6e4000
  2025-07-23T07:27:23.880Z SymBacktrace[11] 0000030efb18aa30 rip=000000596d7fcef7 in function (null) in object /usr/lib/vmware/vsan/bin/clomd loaded at 000000596d6e4000
  2025-07-23T07:27:23.880Z SymBacktrace[12] 0000030efb18aa50 rip=000000596d7fdb9c in function (null) in object /usr/lib/vmware/vsan/bin/clomd loaded at 000000596d6e4000
  2025-07-23T07:27:23.880Z SymBacktrace[13] 0000030efb18aad0 rip=000000596d6fdd72 in function (null) in object /usr/lib/vmware/vsan/bin/clomd loaded at 000000596d6e4000
  2025-07-23T07:27:23.880Z SymBacktrace[14] 0000030efb18ac40 rip=00000059af36fd5d in function __libc_start_main in object /lib64/libc.so.6 loaded at 00000059af34e000
  2025-07-23T07:27:23.880Z SymBacktrace[15] 0000030efb18ad00 rip=000000596d6fe86d in function (null) in object /usr/lib/vmware/vsan/bin/clomd loaded at 000000596d6e4000
  2025-07-23T07:27:23.880Z SymBacktrace[16] 0000030efb18ad08 rip=0000000000000000
  2025-07-23T07:27:23.880Z Failed to dump core: Failure.
  2025-07-23T07:27:23.881Z Msg_Post: Error
  2025-07-23T07:27:23.881Z [msg.log.error.unrecoverable] vSAN Cluster level Object Manager unrecoverable error: (host-#########)
  2025-07-23T07:27:23.881Z NOT_REACHED ####/###/####/vsan_config_builder.c:744
  2025-07-23T07:27:23.881Z [msg.panic.requestSupport.withoutLog] You can request support.
  2025-07-23T07:27:23.881Z [msg.panic.requestSupport.vmSupport.vmx86]
  2025-07-23T07:27:23.881Z To collect data to submit to VMware technical support, run "vm-support".
  2025-07-23T07:27:23.881Z [msg.panic.response] We will respond on the basis of your support entitlement.
  2025-07-23T07:27:23.881Z ----------------------------------------
  2025-07-23T07:27:23.881Z Exiting
  

 

Environment

VMware vSAN 7.0.x

Cause

This issue happens when the object format flag is stuck on the object(s) where the object format upgrade was running at some point and did not complete or was cancelled using custom scripts.

This made the object configuration in an invalid state. So whenever the clom service is started and when clom tries to process that object, the service crashes because of invalid configuration of the object.

Resolution

The resolution requires running a custom script to clean up the object so that a format change can be completed. 

Please contact Broadcom support to assist.