Panic Memory exceeds hard limit | VPXD Service Crashes Unexpectedly

Products

VMware vCenter Server

Issue/Introduction

VPXD service experiences a crash as it runs out of memory.
You may find core.vpxd-worker.xxxx in the /var/core directory.

You may see the following log snippets in /var/log/vmware/vpxd/vpxd.log:

YYYY-MM-DDTHH:MM:SS.280Z error vpxd[3361455] [Originator@6876 sub=Memory checker] Current value 7870200 exceeds hard limit 7852032. Shutting down process.
YYYY-MM-DDTHH:MM:SS.464Z panic vpxd[3361455] [Originator@6876 sub=Default]
-->
--> Panic: Memory exceeds hard limit. Panic
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 8.0.2, build: build-22617221, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0053B757]
--> backtrace[01] libvmacore.so[0x0043097E]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x00443252]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvmacore.so[0x00515A74]
--> backtrace[04] libvmacore.so[0x00515B8C]: Vmacore::PanicExit(char const*)
--> backtrace[05] libvmacore.so[0x004306A0]: Vmacore::System::ResourceChecker::DoCheck()
--> backtrace[06] libvmacore.so[0x003830E1]
--> backtrace[07] libvmacore.so[0x0037CBE6]
--> backtrace[08] libvmacore.so[0x003824F9]
--> backtrace[09] libvmacore.so[0x0051C093]
--> backtrace[10] libpthread.so.0[0x00008EAE]
--> backtrace[11] libc.so.6[0x000FDE2F]
--> backtrace[12] (no module)
--> [backtrace end]
(END)

Environment

vCenter Server 8.0

Cause

This problem arises from the creation of multiple container views by either the monitoring or backup tool, which are subsequently not properly removed/destroyed.

Resolution

Engage the respective application vendor, identified to be creating those ContainerView objects, for investigation.

Additional Information

Command to check the number of containerview created and the corresponding Session ID, here in this example, we can see that session "525ebb59-96d6-00eb-2ff5-7027e5c1514f" has created "48268" ContainerViews

root@vCSA [ /var/log/vmware/vpxd ]# zgrep vim.view.ViewManager.createContainerView vpxd-*.log.gz | grep BEGIN | awk '{print$16}' | sort | uniq -c | sort -nr |head
  48268 525ebb59-96d6-00eb-2ff5-7027e5c1514f(52b069fa-3396-755a-f828-7487ccc2a6ce)
  45860 52959d32-cb77-eba4-6367-b2e4a0d3ec68(52d70d3f-6814-0af9-2a10-ee17001cdbb0)
  45356 52c3acdb-1df1-885b-8727-76c51b7c9a4d(526108b5-8f80-f727-62c7-415723720f14)
  45067 522c370f-3106-894a-4e6d-5286ecf4af5d(520f4adc-1227-e7be-3ebd-63f9d8ddc8db)
  44735 523cf957-c329-d948-80eb-20ef9e6226e2(52e5c446-bb2e-36dc-7748-3cc6b641ca0a)
  40999 52921754-4c7f-e955-82b3-b312fa38dcaa(52640f48-521c-3fc9-e358-27fdd17bd485)
  30771 5252931e-90b0-619b-ad94-d4c6415e0463(52ffc645-d5b3-a1cf-327e-55a809b9e1ca)
   8805 525b6e2e-6b5f-ced1-4b90-484e6cf953e8(52df7e28-0076-24bb-3df7-1cbb3aca2466)
   6792 5292b233-e171-e24c-0ab8-ec1b8a2d725b(526814f3-683d-c905-2675-1044ccf0850f)
   4120 52bf2a37-f3f0-b985-6246-d2cac5b37f40(52df75f8-8ef8-6916-314a-6921f8a77f34)

However there is no trace of the ContainerViews being destroyed for the given Session ID.

root@vCSA [ /var/log/vmware/vpxd ]# zgrep vim.view.View.destroy  vpxd-*.log.gz | grep BEGIN | awk '{print$16}' | sort | uniq -c | sort -nr |head
   8801 525b6e2e-6b5f-ced1-4b90-484e6cf953e8(52df7e28-0076-24bb-3df7-1cbb3aca2466)
   6788 5292b233-e171-e24c-0ab8-ec1b8a2d725b(526814f3-683d-c905-2675-1044ccf0850f)
   4116 52bf2a37-f3f0-b985-6246-d2cac5b37f40(52df75f8-8ef8-6916-314a-6921f8a77f34)
   3787 52065207-4199-556d-2e45-e2b974c2c5fa(52822f00-bb09-f994-edbe-a7b9357f4504)
   3756 52548dba-f5e3-9a1c-15bf-8a249d01aabf(523cf599-063f-4c4a-87fd-fd8dca297686)
   3516 52474f2f-aa8b-6a37-1f89-b9c8c6a12a1e(525491a5-1503-48c3-a804-a7dce5d35f7d)
   2232 52f8641d-e7a9-ef01-270e-de8a6a20668b(52b83d28-7f99-0ca4-1232-e183a3fb4cc3)
   1697 526cb968-1ca8-4ed1-b2e5-e9f18bc9bff8(52e03950-a039-5e89-d077-87e26fd01073)
   1355 529ec571-32ce-9b4a-dbbf-f5f7c6f4def3(52ddf09a-a987-0992-7149-455d131377d7)
   1030 522498ac-239e-e01f-a954-671e1da2014a(52f22e9f-43c5-3e60-ebf8-10c43dc2cc64)

The given Session ID can be utilized (in the vpxd-profiler.log) for tracking the IP address responsible for generating the containerviews.

zgrep -i 525ebb59-96d6-00eb-2ff5-7027e5c1514f vpxd-profiler-*.log.gz


vpxd-profiler-178.log.gz:--> /SessionStats/SessionPool/Session/Id='525ebb59-96d6-00eb-2ff5-7027e5c1514f'/Username='VSPHERE.LOCAL\Administrator'/ClientIP=x.x.x.x'/PropertyCollector/QueuedOpsCount/total 0
vpxd-profiler-178.log.gz:--> /SessionStats/SessionPool/Session/Id='525ebb59-96d6-00eb-2ff5-7027e5c1514f'/Username='VSPHERE.LOCAL\Administrator'/ClientIP='x.x.x.x'/PropertyCollector/TriggeredFiltersCount/total 0
vpxd-profiler-178.log.gz:--> /SessionStats/SessionPool/Session/Id='525ebb59-96d6-00eb-2ff5-7027e5c1514f'/Username='VSPHERE.LOCAL\Administrator'/ClientIP='x.x.x.x'/PropertyCollector/NullCollectorCount/total 0