vpxd crashes when using third party monitoring tool Opmanager.
Article ID: 367047
Updated On:
Products
Issue/Introduction
1. The vpxd service crashes on a regular basis when memory exceeds hard limit.
2. vpxd cores are generated in /storage/core.
3. The vpxd log (/var/log/vmware/vpxd) from the time of the crash will be similar to the following:
yyyy-mm-dd error vpxd[38084] [Originator@6876 sub=Memory checker] Current value 4036720 exceeds hard limit 4036608. Shutting down process.
yyyy-mm-dd panic vpxd[38084] [Originator@6876 sub=Default]
-->
--> Panic: Memory exceeds hard limit. Panic
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 7.0.3, build: build-24026615, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0037DB8B]
--> backtrace[01] libvmacore.so[0x002C79C5]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x002D6C5B]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvmacore.so[0x00370CD7]
--> backtrace[04] libvmacore.so[0x00370DF3]: Vmacore::PanicExit(char const*)
--> backtrace[05] libvmacore.so[0x002C7827]: Vmacore::System::ResourceChecker::DoCheck()
--> backtrace[06] libvmacore.so[0x0023B390]
--> backtrace[07] libvmacore.so[0x002349E7]
--> backtrace[08] libvmacore.so[0x00239F75]
--> backtrace[09] libvmacore.so[0x003765C0]
--> backtrace[10] libpthread.so.0[0x00007F87]
--> backtrace[11] libc.so.6[0x000F36BF]
--> backtrace[12] (no module)
--> [backtrace end]
4. Throughout the same log (and historical vpxd logs), there will be many performance manager/container view tasks, similar to the following (output below snipped for readability):
View.ViewManager.createContainerView -- 521####-####-####-####-########65(52####-####-=####-####-####a5)
PerfMgr -- vim.PerformanceManager.queryProviderSummary -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)
PerfMgr -- vim.PerformanceManager.queryStats -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)
ViewManager -- vim.view.ViewManager.createContainerView -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)
PerfMgr -- vim.PerformanceManager.queryProviderSummary -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)
PerfMgr -- vim.PerformanceManager.queryStats -- 52####-####-####-####-1#####e65(52####-####-###-###-######a5)
Note that all these tasks are from the same session ID: 52####-####-####-####-1#####e65
5. Using that session ID, grep in the vpxd-profiler logs to match it to a username/IP address, for example:
# grep 52####-####-####-####-1#####e65 vpxd-profiler-* | grep ClientIP | awk -F / '{print $5 $6 $7}' | sort -u
Example output from the above grep command:
Id='52####-####-####-####-1#####e65' Username='DOMAIN\USERNAME' ClientIP='###.###.###,###'
Note:
Where 52###-###-####-###e65 (Id) contains the IP address of the server running Opmanager. DOMAIN\USERNAME is the AD domain and Opmanager service username.
Environment
- vCenter Server Appliance 7.x
- vCenter Server Appliance 8.x
Cause
- Opmanager (3rd party monitoring) is in use.
Resolution
Contact Opmanager vendor for updates.
Workaround:
Stop (temporarily) Opmanager until vendor updates can be applied.
Additional Information
Feedback
