The vpxd service crashes intermittently in vCenter when the Dell OMEVV plugin is enabled

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

Messages in /var/log/vmware/vpxd/vpxd.log might show some specific sessions sending blocking calls to the property collector.

2024-11-21T17:15:03.732-05:00 warning vpxd[59087] [Originator@6876 sub=PropertyCollector opID=717a7e91] Session XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXabcd has issued 18 concurrent blocking calls to the property collector. This is almost certainly an error in the logic of the client application.
2024-11-21T17:15:03.732-05:00 warning vpxd[59533] [Originator@6876 sub=PropertyCollector opID=320ef7a2] Session XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXabcd has issued 19 concurrent blocking calls to the property collector. This is almost certainly an error in the logic of the client application.
2024-11-21T17:15:03.732-05:00 warning vpxd[58065] [Originator@6876 sub=PropertyCollector opID=437405cc] Session XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXabcd has issued 20 concurrent blocking calls to the property collector. This is almost certainly an error in the logic of the client application.
2024-11-21T17:15:03.732-05:00 warning vpxd[59868] [Originator@6876 sub=PropertyCollector opID=67b1cf0a] Session XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXabcd has issued 21 concurrent blocking calls to the property collector. This is almost certainly an error in the logic of the client application.

The session ID from the above messages is found in vpxd-profiler log with an IP and Username belonging to the OMEVV Plugin

$ grep "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXabcd" var/log/vmware/vpxd/vpxd-profiler-*.log
/SessionStats/SessionPool/Session/Id='XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXabcd'/Username='VSPHERE.LOCAL\EXAMPLEOMEVVPLUGINUSER'/ClientIP='10.10.10.10'/PropertyCollector/TriggeredFiltersCount/total 19

 

Eventually, the vpxd service will crash from memory exhaustion.

2024-11-21T16:43:49.702-05:00 error vpxd[21015] [Originator@6876 sub=Memory checker] Current value 10551476 exceeds hard limit 10549184. Shutting down process.
2024-11-21T16:43:49.936-05:00 panic vpxd[21015] [Originator@6876 sub=Default]
-->
--> Panic: Memory exceeds hard limit. Panic
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 7.0.3, build: build-24322018, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0037DB8B]
--> backtrace[01] libvmacore.so[0x002C79C5]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x002D6C5B]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvmacore.so[0x00370CD7]
--> backtrace[04] libvmacore.so[0x00370DF3]: Vmacore::PanicExit(char const*)
--> backtrace[05] libvmacore.so[0x002C7827]: Vmacore::System::ResourceChecker::DoCheck()
--> backtrace[06] libvmacore.so[0x0023B390]
--> backtrace[07] libvmacore.so[0x002349E7]
--> backtrace[08] libvmacore.so[0x00239F75]
--> backtrace[09] libvmacore.so[0x003765C0]
--> backtrace[10] libpthread.so.0[0x00007F87]
--> backtrace[11] libc.so.6[0x000F36BF]
--> backtrace[12] (no module)
--> [backtrace end]

Note that these lines are common to all vpxd memory exhaustion crashes.

 

Environment

• vCenter Server 7.0
• vCenter Server 8.0

Cause

The Dell OMEVV plugin has been found to incorrectly create property collector filters in vpxd without destroying stale or unused ones.

Resolution

Please work with Dell to resolve this plugin issue permanently. 

To work around the issue, disable the Dell OMEVV plugin in vCenter under Administration > Solutions.