vmware-vpxd service crash with core dump "core.vpxd-worker.xxxxxxx" on vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

• You will observe core dumps generated for each crash in the /var/core  core.vpxd-worker.xxxxxxx.

• /var/log/vmware/vpxd/vpxd.log will contains snippets similar to below,

 

     YYYY-MM-DDThh:mm:ss panic vpxd[08492] [Originator@6876 sub=Default]
         -->
         --> Panic: Memory exceeds hard limit. Panic
         --> Backtrace:
         --> [backtrace begin] product: VMware VirtualCenter,version: 8.0.3, build: build-24674346,, tag: vpxd, cpu: x86_64, os: linux, buildType: release
         --> backtrace[00] libvmacore.so[0x0037DB8B]
         --> backtrace[01] libvmacore.so[0x002C79C5]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
         --> backtrace[02] libvmacore.so[0x002D6C5B]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
         --> backtrace[03] libvmacore.so[0x00370CD7]
         --> backtrace[04] libvmacore.so[0x00370DF3]: Vmacore::PanicExit(char const*)
         --> backtrace[05] libvmacore.so[0x002C7827]: Vmacore::System::ResourceChecker::DoCheck()
         --> backtrace[06] libvmacore.so[0x0023B390]
         --> backtrace[07] libvmacore.so[0x002349E7]
         --> backtrace[08] libvmacore.so[0x00239F75]
         --> backtrace[09] libvmacore.so[0x003765C0]
         --> backtrace[10] libpthread.so.0[0x00007F87]
         --> backtrace[11] libc.so.6[0x000F36BF]
         --> backtrace[12] (no module)
         --> [backtrace end]

 

• /var/log/vmware/vmon/vmon.log

YYYY-MM-DDThh:mm:ss In(05) host-1654265 Notify vMon about vpxd-worker dumping core. Pid : 1XXXXX7
YYYY-MM-DDThh:mm:ss In(05) host-1654265 Successfully notified vMon.
YYYY-MM-DDThh:mm:ss Wa(03) host-1654265 Not enough canditate files found to free up core disk space.
YYYY-MM-DDThh:mm:ss Er(02) host-1654265 Write failed. Err: Success

Environment

VMware vCenter 8.x

Cause

Heap memory of vCenter vpxd gets exhausted 

Resolution

Increase vmware-vpxd heap Memory:

• Allocate additional memory to the VCSA to accommodate all incoming requests.

• Reference KB 320871 for instructions on manually increasing heap memory on vCenter.

Note:

• The above is just a workaround however, you will need to check if there are multiple sessions/api calls reaching the vCenter from any monitoring or third party tools and reduce the sessions