VLCM service will not start

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

VCLM is stopped and cannot be started and trying to start the service manually with show 

"an error occurred while starting service '%(0)'"

Environment

VCF 5.x

SDDC 5.2.

vCenter 8.x

Cause

Underlying file system corruption for workflow_State.json files that VLCM requires for healthy startup 

var/log/vmware/vlcm/vlcm.log will show


xxxx-xx-xxTxx:xx:xx.xxxZ error vlcm [db/file.go:55] wfstatedb: failed to read workflow engine database /storage/lifecycle/vlcm/upgrade/db/current/workflow_state.json: open /storage/lifecycle/vlcm/upgrade/db/current/workflow_state.json: structure needs cleaning
xxxx-xx-xxTxx:xx:xx.xxxZ error vlcm [upgrade/controller_factory.go:56] Create NDU upgrade controller failed
xxxx-xx-xxTxx:xx:xx.xxxZ fatal vlcm [cmd/main.go:234] Failed to initialize service map: open /storage/lifecycle/vlcm/upgrade/db/current/workflow_state.json: structure needs cleaning

When the files is reviewed manually, operation on the file and datastore hang 

Trying to list files in the location will return

"ls: cannot access 'workflow_state.json': structure needs clearning

total 0

?????????????????????? workflow_state.json

 

Resolution

Use Option 2 in the kb bellow to repair the effected filesystems

https://knowledge.broadcom.com/external/article/326323/failed-to-start-file-system-check-on-dev.html

If this does not resolve the issue open a case with Broadcom Support