/storage/core partition full due to core files in vCenter Server Appliance

Products

• VMware vCenter Server

Issue/Introduction

• A common issue observed in vCenter Server Appliances is the /storage/core partition filling up. This occurs due to the generation of numerous core dump files, such as core.netlogond.####, core.vpxd-worker.####, core.updatemgr-worker.#####.
• The vmware-vpxd service goes to stopped state:

root@[ /storage/core ]# service-control -status -all
Running:
applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-ce
rtificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-hvc vmware-infraprofile vmware-p
erfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sps vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware
-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp
Stopped:
vmcam wmonapi vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-vcha vmware-vpxd

• While the exact partition size varies by VCSA deployment option, this problem can affect any sized appliance, leading to critical disk space exhaustion.
• Error: File system/storage/core is low on storage space. Increase the size of disk /storage/core:

Below is the output from vCenter cli:

[ /storage/core ]# df -h /storage/core/
Filesystem                            Size   Used   Avail   Use%   Mounted on
/dev/mapper/core_vg-core   98G    74G    20G    80%     /storage/core

Sample Core files :

[ /storage/core ]# du -sh *
16M  core.ps.######
16M  core.ps.######
16M  core.ps.######
16M  core.ps.######
16M  core.ps.######
16M  core.ps.######
175M core.python.######
175M core.python.######
32G  core.updatemgr-worke.#######
34G  core.updatemgr-worke.#######
7.8G core.vpxd-worker.####
16K  lost+found
4.0K netdumps
20K  software-update
4.0K vmware-wcp

Environment

VMware vCenter Server Appliance 8.x

Cause

This occurs due to the repeated generation of core dump files, commonly named:

• core.netlogond.######

• core.updatemgr-worker.#####

• Other service-related core files

These files are typically created when internal VCSA services crash or encounter fatal errors.

Resolution

Workaround:

• To free up disk space, manually delete core files located in /storage/core using the rm -f command.

For example, [ /storage/core ]# rm -rf core.updatemgr-worker.######

For exmaple, [ /storage/core ]# rm -rf core.vpxd-worker.######

• Verify available storage after removing files from core partition:

[ /storage/core ]# df -h /storage/core/

Filesystem                 Size  Used   Avail Use%   Mounted on
/dev/mapper/core_vg-core   98G   8.2G   85G   9%     /storage/core

 

Please Note:

1. A full vCenter File based backup and Offline Snapshots of the vCenter is mandatory before we execute the rm -rf command 
2. "rm -rf is a destructive command has to be used cautiously and Broadcom is not responsible for any loss of data by incorrectly executing the above mentioned commands.
Below is the KB for reference :
https://knowledge.broadcom.com/external/article/326764