vCenter Server vpxd service crashes after deploying or re-synchronizing VMware HCX Service Mesh

Products

• VMware HCX
• VMware vCenter Server

Issue/Introduction

• The vCenter Server vpxd service crashes frequently after you deploy or resynchronize an HCX service mesh.
• You are using "Pure1 VM Analytics Collector"
• When you review the vcenter /var/log/vmware/vpxd/vpxd.log, you see entries similar to the following:

  error vpxd[####] [Originator@6876 sub=Memory checker] Current value 33743732 exceeds hard limit 33554432. Shutting down process. panic vpxd[####] [Originator@6876 sub=Default] --> Panic: Memory exceeds hard limit.

• Run the following command in the vCenter to identify the client IPs and user account creating the most number of connections. 

   grep "ClientIP" /var/log/vmware/vpxd/vpxd-profiler*.log | grep -v "127.0.0.1" | awk -F"'" '{print $6, $4}' | sort | uniq -c | sort -nr | head -n 10

Environment

VMware vCenter Server
VMware HCX

Cause

This issue occurs because the "Pure1 VM Analytics Collector" overwhelms the vCenter Server with an excessive volume of sessions and property collector requests. The surge in activity typically coincides with HCX service mesh operations, leading to vpxd memory exhaustion and a subsequent service crash.

Resolution

Engage Pure Storage support to review the behavior of the Pure1 automation script. Request a review of how the collector manages sessions and property collection frequency to prevent flooding the vCenter Server.

Workaround: Disable the "Pure1 VM Analytics Collector" before you initiate HCX service mesh deployments or re-synchronizations. You can re-enable the collector once the HCX operations are complete.