Heavy k8s workload caused vsan-health core dump in 80u2.

Products

• VMware vCenter Server

Issue/Introduction

This article helps users to prevent core dumps from happening.

Symptoms:

1. Under heavy workload, vsan-health core dumps are observed.
2. Core dump backtrace shows below trace:

#4 0x00007fd446801e1b in Vmacore::FormatObject (v=0x7fd40c2403f0, w=0x7fd3e25be4b0) at bora/vim/lib/vmacore/main/formatter.cpp:236
...
#19 Vmacore::Service::LogInternal (log=0x7fd4103ba6d0, n=n@entry=Vmacore::Service::Logger::info, m=m@entry=0x7fd438d54f57 "invoking next workflow %1 pending on resource %2, activation %3.", f1=..., f2=..., f3=...) at bora/vim/lib/vmacore/main/logger.cpp:1940

Environment

VMware vCenter Server 8.0.2

Cause

CNS logging has a race condition when processing requests.

Resolution

VMware is aware of this issue and working to resolve this in a future release.

Workaround:
The logging path is touched only by CNS new serialization feature in 80U2, which reduces unnecessary calls to vm when multiple volumes are attached to it. Customer can avoid core dumps by disabling cns new serialization in below steps:

1. open /usr/lib/vmware-vsan/VsanVcMgmtConfig.xml on vc.
2. change below config to false.<newSerializationEnabled>true</newSerializationEnabled>
3. restart vsan-health with 'vmon-cli -r vsan-health'.