VPXD service on vCenter Server Appliance crashes during backup
Article ID: 313932
Updated On:
Products
VMware vCenter Server
Issue/Introduction
Symptoms:
- VPXD service crashes during backup via third-party backup solution
- The issue occurs after upgrade of software by that solution
In the vpxd.log file, you see entries similar to (Avamar used as example):
2020-03-19T14:44:16.762Z info vpxd[43837] [Originator@6876 sub=ViewManager opID=141e919a] Duplicate container view for Object : group-d1 [
--> user - VSPHERE.LOCAL\Avamar
--> session - 522ed27e-7545-7381-5d0c-612dfb07161e
--> IP - 10.x.x.x
--> Number of duplicates - 50567]
2020-03-19T14:44:16.764Z info vpxd[43837] [Originator@6876 sub=vpxLro opID=141e919a] [VpxLRO] -- FINISH lro-1721828
2020-03-19T14:44:16.803Z info vpxd[43913] [Originator@6876 sub=ViewManager opID=362b12b3] Duplicate container view for Object : group-d1 [
--> user - VSPHERE.LOCAL\Avamar
--> session - 522ed27e-7545-7381-5d0c-612dfb07161e
--> IP - 10.x.x.x
--> Number of duplicates - 50568]
2020-03-19T14:44:16.819Z info vpxd[43913] [Originator@6876 sub=vpxLro opID=362b12b3] [VpxLRO] -- FINISH lro-1721860
2020-03-19T14:44:16.847Z info vpxd[43839] [Originator@6876 sub=ViewManager opID=7677d890] Duplicate container view for Object : group-d1 [
--> user - VSPHERE.LOCAL\Avamar
--> session - 522ed27e-7545-7381-5d0c-612dfb07161e
--> IP - 10.2.x.x
--> Number of duplicates - 50569]
2020-03-19T14:44:16.985Z error vpxd[43808] [Originator@6876 sub=Memory checker] Current value 3972748 exceeds hard limit 3961856. Shutting down process.
2020-03-19T14:44:17.099Z panic vpxd[43808] [Originator@6876 sub=Default]
-->
--> Panic: Memory exceeds hard limit. Panic
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.7.0, build: build-14792544, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B00F8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B2624]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x002AE2CE]
--> backtrace[03] libvmacore.so[0x002AE3AE]: Vmacore::PanicExit(char const*)
--> backtrace[04] libvmacore.so[0x001B6242]: Vmacore::System::ResourceChecker::DoCheck()
--> backtrace[05] libvmacore.so[0x0027F095]
--> backtrace[06] libvmacore.so[0x00236F71]
--> backtrace[07] libvmacore.so[0x0023723A]
--> backtrace[08] libvmacore.so[0x002B569D]
--> backtrace[09] libpthread.so.0[0x000073D4]
--> backtrace[10] libc.so.6[0x000E8EDD]
--> [backtrace end]
--> user - VSPHERE.LOCAL\Avamar
--> session - 522ed27e-7545-7381-5d0c-612dfb07161e
--> IP - 10.x.x.x
--> Number of duplicates - 50567]
2020-03-19T14:44:16.764Z info vpxd[43837] [Originator@6876 sub=vpxLro opID=141e919a] [VpxLRO] -- FINISH lro-1721828
2020-03-19T14:44:16.803Z info vpxd[43913] [Originator@6876 sub=ViewManager opID=362b12b3] Duplicate container view for Object : group-d1 [
--> user - VSPHERE.LOCAL\Avamar
--> session - 522ed27e-7545-7381-5d0c-612dfb07161e
--> IP - 10.x.x.x
--> Number of duplicates - 50568]
2020-03-19T14:44:16.819Z info vpxd[43913] [Originator@6876 sub=vpxLro opID=362b12b3] [VpxLRO] -- FINISH lro-1721860
2020-03-19T14:44:16.847Z info vpxd[43839] [Originator@6876 sub=ViewManager opID=7677d890] Duplicate container view for Object : group-d1 [
--> user - VSPHERE.LOCAL\Avamar
--> session - 522ed27e-7545-7381-5d0c-612dfb07161e
--> IP - 10.2.x.x
--> Number of duplicates - 50569]
2020-03-19T14:44:16.985Z error vpxd[43808] [Originator@6876 sub=Memory checker] Current value 3972748 exceeds hard limit 3961856. Shutting down process.
2020-03-19T14:44:17.099Z panic vpxd[43808] [Originator@6876 sub=Default]
-->
--> Panic: Memory exceeds hard limit. Panic
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.7.0, build: build-14792544, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B00F8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B2624]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x002AE2CE]
--> backtrace[03] libvmacore.so[0x002AE3AE]: Vmacore::PanicExit(char const*)
--> backtrace[04] libvmacore.so[0x001B6242]: Vmacore::System::ResourceChecker::DoCheck()
--> backtrace[05] libvmacore.so[0x0027F095]
--> backtrace[06] libvmacore.so[0x00236F71]
--> backtrace[07] libvmacore.so[0x0023723A]
--> backtrace[08] libvmacore.so[0x002B569D]
--> backtrace[09] libpthread.so.0[0x000073D4]
--> backtrace[10] libc.so.6[0x000E8EDD]
--> [backtrace end]
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment
Environment
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 7.0.3.x
Cause
This issue is occurring due to duplicate container views being created during backup and then not being destroyed.
Resolution
If using Avamar, engage Avamar support as the issue is specific to Avamar version 18.1 and 18.2
Note that other vendors may also be affected. If the symptoms and logs match per that vendor, engage their support accordingly.Workaround:
As a temporary workaround, do not run multiple concurrent backup jobs.
Additional Information
For more information refer to After upgrading to Avamar 18.1/18.2 vCenter VPXD service is constantly crashing
Feedback
Yes
No
Powered by
