High memory consumption by the "nsx-edge-exporter" service resulting in high memory usage on the NSX Edges.
search cancel

High memory consumption by the "nsx-edge-exporter" service resulting in high memory usage on the NSX Edges.

book

Article ID: 411735

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • We may observe high memory consumption by the nsx-edge-exporter service, leading to overall high memory usage on the NSX Edges.



  • Review of /var/log/syslog on the problematic Edge reveals frequent occurrences of Invalid corealtionId messages and PurgeStaleCallHandles entries, indicating issues with handling inter-process communication (RPC) calls:

    syslog:35:2025-08-25T15:11:09.074Z NSX XXXX829 - [nsx@6876 comp="nsx-edge" subcomp="mpa-client" tid="XXXX908" level="INFO"] [AggSvc-Lb] MpaClientAppSpecific::RespondMsg corelationId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX1fee)
    syslog:37:2025-08-25T15:11:09.074Z NSX XXXX829 - [nsx@6876 comp="nsx-edge" subcomp="mpa-client" tid="XXXX908" level="INFO"] [AggSvc-Lb] MpaClientAppSpecific::RespondMsg Invalid corealtionId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX1fee) and empty amqpRoutingKey
    syslog:132:2025-08-25T15:11:09.656Z NSX XXXX829 - [nsx@6876 comp="nsx-edge" subcomp="mpa-client" tid="XXXX908" level="INFO"] [AggSvc-Lb] MpaClientAppSpecific::RespondMsg corelationId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXac0c)
    syslog:133:2025-08-25T15:11:09.656Z NSX XXXX829 - [nsx@6876 comp="nsx-edge" subcomp="mpa-client" tid="XXXX908" level="INFO"] [AggSvc-Lb] MpaClientAppSpecific::RespondMsg Invalid corealtionId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXac0c) and empty amqpRoutingKey
    [....]
    syslog.1:588040:2025-08-25T14:43:08.511Z NSX XXXX829 - [nsx@6876 comp="nsx-edge" subcomp="mpa-client" tid="XXXX922" level="INFO"] [AggSvc-Lb] MpaClientHostService Received: Msg type (com.vmware.nsx.management.agg.messaging.ClientDataRequestMsg)corelationId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX1fee) trackingId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX1fee) from APH (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX84a6)
    syslog.1:588075:2025-08-25T14:43:09.876Z NSX XXXX829 - [nsx@6876 comp="nsx-edge" subcomp="mpa-client" tid="XXXX922" level="INFO"] [AggSvc-Lb] MpaClientHostService Received: Msg type (com.vmware.nsx.management.agg.messaging.OnDemandStatsRequestMsg)corelationId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXac0c) trackingId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXac0c) from APH (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX2e86)
    syslog.1:632747:2025-08-25T14:52:24.776Z NSX XXXX829 - [nsx@6876 comp="nsx-edge" subcomp="mpa-client" tid="XXXX922" level="INFO"] [AggSvc-Lb] MpaClientAppSpecific PurgeStaleCallHandles Removing call handle, corelationId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXac0c)
    syslog.1:632750:2025-08-25T14:52:24.776Z NSX XXXX829 - [nsx@6876 comp="nsx-edge" subcomp="mpa-client" tid="XXXX922" level="INFO"] [AggSvc-Lb] MpaClientAppSpecific PurgeStaleCallHandles Removing call handle, corelationId (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX1fee)

Environment

VMware NSX

Cause

This issue occurs when the Management Plane (MP) generates a high volume of requests to the Load Balancer (LB) vertical thread on the NSX Edge. The LB vertical thread, which is responsible for retrieving statistics from both LB Nginx and the datapath, may not be able to process all these requests in a timely manner.

When requests cannot be handled promptly, they might time out. The system then purges the corresponding call handles (e.g., via PurgeStaleCallHandles) to release resources. However, if the LB vertical thread eventually attempts to send a response for an already-purged correlation ID, it results in an Invalid corealtionId error and a failure to send the message. This cycle of delayed processing, premature purging, and failed responses leads to the accumulation of RPC calls and increased memory usage by the nsx-edge-exporter service.

Resolution

To resolve the root cause, reduce the no. of queries being sent to fetch LB status and statistics. If this is not an option, follow the below:

  • Keep monitoring the memory usage of the edge-exporter service. When the edge's overall memory usage reaches 90p and the edge-exporter service has considerable contribution, restart the edge-exporter service using the command "systemctl status nsx-edge-exporter".
  • The issue with the graceful cleanup of memory post the messaging handling failures is scheduled to be fixed in one of the upcoming release.
Powered by Wolken Software