vCenter keeps on crashing intermittently

Products

• VMware vSphere ESXi

Issue/Introduction

VPXD crashes with vpxd core dumps with the header "Panic: Memory exceeds hard limit. Panic"

/var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDTHH:MM:SS.sssZ error vpxd[XXXXX] [Originator@XXXX sub=Memory checker] Current value 22595688 exceeds hard limit 22528000. Shutting down process.
YYYY-MM-DDTHH:MM:SS.sssZ info vpxd[XXXXX] [Originator@XXXX sub=vpxLro opID=XXXXXXXX] [VpxLRO] -- FINISH lro-XXXXXXX
YYYY-MM-DDTHH:MM:SS.sssZ info vpxd[XXXXX] [Originator@XXXX sub=vpxLro opID=XXXXXXXX] [VpxLRO] -- BEGIN lro-XXXXXX -- guestOperationsProcessManager -- vim.vm.guest.ProcessManager.listProcesses -- <Process List UUID>
YYYY-MM-DDTHH:MM:SS.sssZ info vpxd[XXXXX] [Originator@XXXX sub=vpxLro opID=XXXXXXXX] [VpxLRO] -- FINISH lro-XXXXXXX
YYYY-MM-DDTHH:MM:SS.sssZ panic vpxd[XXXXX] [Originator@XXXX sub=Default]
   -->
   --> Panic: Memory exceeds hard limit. Panic
   --> Backtrace:
   --> [backtrace begin] product: VMware VirtualCenter, version: 7.0.3, build: build-22357613, tag: vpxd, cpu: x86_64, os: linux, buildType: release
   --> backtrace[00] libvmacore.so[0x00XXXXXX]
   --> backtrace[01] libvmacore.so[0x00XXXXXX]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
   --> backtrace[02] libvmacore.so[0x00XXXXXX]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
   --> backtrace[03] libvmacore.so[0x00XXXXXX]
   --> backtrace[04] libvmacore.so[0x00XXXXXX]: Vmacore::PanicExit(char const*)
   --> backtrace[05] libvmacore.so[0x00XXXXXX]: Vmacore::System::ResourceChecker::DoCheck()
   --> backtrace[06] libvmacore.so[0x00XXXXXX]
   --> backtrace[07] libvmacore.so[0x00XXXXXX]
   --> backtrace[08] libvmacore.so[0x00XXXXXX]
   --> backtrace[09] libvmacore.so[0x00XXXXXX]
   --> backtrace[10] libpthread.so.0[0x00XXXXXX]
   --> backtrace[11] libc.so.6[0x00XXXXXX]

Environment

VMware vCenter Server Version: 7.0.3

 

Cause

The vpxd service will randomly stop running and the above backtrace will be logged in the vpxd logs. This is caused by a process or application that is causing the vpxd service to run out of memory. 

Resolution

This could be caused due to memory exhaustion due to a process or application which is frequently querying vpxd service. To identify the process or application, please reach out to Broadcom Technical Support Team.