vCenter Server Appliance /storage/core partition full due to core files

Products

• VMware vCenter Server

Issue/Introduction

• The vmware-vpxd service goes to stopped state.

root@[ /storage/core ]# service-control -status -all
Running:
applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-hvc vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sps vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp
Stopped:
vmcam wmonapi vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-vcha vmware-vpxd

• While the exact partition size varies by VCSA deployment option, this problem can affect any sized appliance, leading to critical disk space exhaustion.
  Error: File system/storage/core is low on storage space. Increase the size of disk /storage/core
  
  
• Below is the output from the vCenter SSH session:

[ /storage/core ]# df -h /storage/core/
Filesystem                            Size   Used   Avail   Use%   Mounted on
/dev/mapper/core_vg-core   98G    74G    20G    80%     /storage/core

Sample Core files:

[ /storage/core ]# du -sh *
16M  core.ps.######
16M  core.ps.######
16M  core.ps.######
16M  core.ps.######
16M  core.ps.######
16M  core.ps.######
175M core.python.######
175M core.python.######
32G  core.updatemgr-worker.#######
34G  core.updatemgr-worker.#######
7.8G core.vpxd-worker.####
16K  lost+found
4.0K netdumps
20K  software-update
4.0K vmware-wcp

Environment

• VMware vCenter Server Appliance 8.x
• VMware vCenter Server Appliance 7.x

Cause

• This occurs due to the generation of numerous core dump files, such as:
  core.netlogond.######
core.updatemgr-worker.#####
core.vpxd-worker.####
Other service-related core files

• These files are typically created when the vCenter services crash or encounter fatal errors.

Resolution

Note: Take a snapshot of the vCenter server before performing the steps. (VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice)

rm is a destructive command that has to be used cautiously and Broadcom is not responsible for any loss of data by incorrectly executing the above mentioned command. Data recovery services for data not recoverable by VMware Technical Support

1. To free up disk space, manually delete core files located in /storage/core using the rm command.
   Example:
[ /storage/core ]# rm core.updatemgr-worker.######
[ /storage/core ]# rm core.vpxd-worker.######


2. Verify available storage after removing files from core partition:
   [ /storage/core ]# df -h /storage/core/
Filesystem                 Size  Used   Avail Use%   Mounted on
/dev/mapper/core_vg-core   98G   8.2G   85G   9%     /storage/core