Repeated and regular vpxd crashes when using third party monitoring tool Opmanager.

Products

• VMware vCenter Server

Issue/Introduction

1. The vpxd service crashes on a regular basis when memory exceeds hard limit.
2. vpxd cores are generated in /storage/core.
3. The vpxd log (/var/log/vmware/vpxd) from the time of the crash will be similar to the following
   

   yyyy-mm-dd error vpxd[38084] [Originator@6876 sub=Memory checker] Current value 4036720 exceeds hard limit 4036608. Shutting down process.
   yyyy-mm-dd panic vpxd[38084] [Originator@6876 sub=Default]
   -->
   --> Panic: Memory exceeds hard limit. Panic
   --> Backtrace:
   --> [backtrace begin] product: VMware VirtualCenter, version: 7.0.3, build: build-24026615, tag: vpxd, cpu: x86_64, os: linux, buildType: release
   --> backtrace[00] libvmacore.so[0x0037DB8B]
   --> backtrace[01] libvmacore.so[0x002C79C5]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
   --> backtrace[02] libvmacore.so[0x002D6C5B]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
   --> backtrace[03] libvmacore.so[0x00370CD7]
   --> backtrace[04] libvmacore.so[0x00370DF3]: Vmacore::PanicExit(char const*)
   --> backtrace[05] libvmacore.so[0x002C7827]: Vmacore::System::ResourceChecker::DoCheck()
   --> backtrace[06] libvmacore.so[0x0023B390]
   --> backtrace[07] libvmacore.so[0x002349E7]
   --> backtrace[08] libvmacore.so[0x00239F75]
   --> backtrace[09] libvmacore.so[0x003765C0]
   --> backtrace[10] libpthread.so.0[0x00007F87]
   --> backtrace[11] libc.so.6[0x000F36BF]
   --> backtrace[12] (no module)
   --> [backtrace end]

4. Throughout the same log (and historical vpxd logs), there will be many performance manager/container view tasks, similar to the following (output below snipped for readability):
   

   View.ViewManager.createContainerView -- 521####-####-####-####-########65(52####-####-=####-####-####a5)
   
   PerfMgr -- vim.PerformanceManager.queryProviderSummary -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)
   
   PerfMgr -- vim.PerformanceManager.queryStats -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)
   
   ViewManager -- vim.view.ViewManager.createContainerView -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)
   
   PerfMgr -- vim.PerformanceManager.queryProviderSummary -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)
   
   PerfMgr -- vim.PerformanceManager.queryStats -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)

   Note that all these tasks are from the same session ID:
   

   52####-####-####-####-1#####e65

    

5. Using that session ID, grep in the vpxd-profiler logs to match it to a username/IP address, for example:
   

   grep 52####-####-####-####-1#####e65 vpxd-profiler-* | grep ClientIP | awk -F / '{print $5 $6 $7}' | sort -u

6. Example output from the above grep command:
   

   Id='52####-####-####-####-1#####e65'Username='DOMAIN\USERNAME'ClientIP='###.###.####,###'

   Note:

   Where 52###-###-####-###e65 is the IP address of the server running Opmanager and DOMAIN\USERNAME is the AD domain and Opmanager service username.

Environment

• vCenter Server Appliance 7.x
• vCenter Server Appliance 8.x

Cause

• Opmanager (3rd party monitoring) is in use.

Resolution

• To resolve this issue engage Opmanager vendor to get an update on the application.
• Workaround:
  Stop Opmanager temporarily while waiting to update Opmanager software

Additional Information