The vCenter service vpxd crashes with a "Panic: Memory exceeds hard limit." when using the NSX-vSphere Adapter in Aria Operations

Products

• VMware vCenter Server

Issue/Introduction

• One or more Aria Operations instances are accessing the vCenter Server to collect data.
• One or more of these instances have the NSX-vSphere Adapter installed and running.
• The vpxd process in the vCenter Server crashes intermittently or frequently.
• vSphere Client becomes unusable with code "503 Service unavailable" or "no healthy upstream" until the service is restarted.
• In the vpxd logs (/var/log/vmware/vpxd/vpxd-<number>.log), a disproportionately higher number of vim.view.ViewManager.createContainerView tasks than compared to vim.view.View.destroy tasks can be observed.
• Furthermore the vpxd logs show memory exhaustion errors, with messages similar to:

  YYYY-MM-DDThh:mm:ss panic vpxd[#####] [Originator@6876 sub=Default]
           -->
           --> Panic: Memory exceeds hard limit. Panic
           --> Backtrace:
           --> [backtrace begin] product: VMware VirtualCenter, version: 7.0.#, build: build-########, tag: vpxd, cpu: x86_64, os: linux, buildType: release
           --> backtrace[00] libvmacore.so[0x########]
           --> backtrace[01] libvmacore.so[0x########]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
           --> backtrace[02] libvmacore.so[0x########]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
           --> backtrace[03] libvmacore.so[0x########]
           --> backtrace[04] libvmacore.so[0x########]: Vmacore::PanicExit(char const*)
           --> backtrace[05] libvmacore.so[0x########]: Vmacore::System::ResourceChecker::DoCheck()
           --> backtrace[06] libvmacore.so[0x########]
           --> backtrace[07] libvmacore.so[0x########]
           --> backtrace[08] libvmacore.so[0x########]
           --> backtrace[09] libvmacore.so[0x########]
           --> backtrace[10] libpthread.so.0[0x########]
           --> backtrace[11] libc.so.6[0x########]
           --> backtrace[12] (no module)
           --> [backtrace end]

Environment

VMware vCenter Server 7.0.x

Cause

• The NSX-vSphere adapter continuously creates a high volume of ContainerView-based filters in the vpxd process over a short period of time, but fails to destroy them once they are no longer needed.
• Over time, this leads to a buildup of memory usage within the vpxd service. As memory is exhausted, the vpxd process eventually crashes with a panic due to exceeding its memory limits.

Resolution

There is no permanent resolution for this issue due to the underlying design of the NSX-vSphere Adapter.

Workaround:
To mitigate this issue and prevent further crashes of the vpxd service:

• Stop or remove the NSX-vSphere Adapter from all vROps instances connected to the affected vCenter Server.
  Note: The NSX-vSphere Adapter was designed for integration with NSX-V, which is no longer officially supported. For details, refer to End of Availability - VMware NSX Data Center for vSphere
• For environments running NSX-T, use the NSX-T specific adapter provided for vROps, as the NSX-vSphere Adapter is not compatible with NSX-T.

Additional Information

If NSX-V is no longer in use, ensure all associated adapters and integrations are fully removed to prevent service impact.