Agent Memory Usage High is seen in the NSX-T UI Alarm

Products

• VMware NSX

Issue/Introduction

Event ID: transport_node_health.agent_memory_usage_high
Alarm Description:

• Purpose:
  The Alarm for Agent Memory Usage indicates that some of your processes are with high memory usage. When the memory usage of this agent reaches the predefined high threshold the alarm will be raised.
  Currently, the agents on the monitoring list include nsx-cfgagent, nsx-opsagent, nsx-nestdb, nsx-exporter and nsx-vdpi.
  • nsx-cfgagent - Handles the stateful configuration from controller.
  • nsx-opsagent - Handles all stateless operations initiated from NSX manager.
  • nsx-nestdb - Database storing all configurations and runtime states between controller and nsx-cfgagent.
  • nsx-exporter - Responsible for collecting statistics from transport node and sending it to the NSX manager.
  • nsx-vdpi - DFW L7 engine which does deep packet inspections.

Environment

VMware NSX

Resolution

Steps to Resolve

For 4.2.0 and higher

 

Recommended Action:

To avoid unexpected behavior in the monitored agent and process termination due to further increase in the memory usage you will need to try to reduce memory usage using the following procedure.

1. nsx-cfagent:
   • Gather information about operations performed prior to observation of the alarm.
   • If the prior operation was adding more DFW rules and groups, please pause further additions until the alarm is cleared.
   • If workloads were vMotioned onto the host prior, please consider moving those workloads to less loaded hosts.
   • If the alarm persists, please consider moving some workloads to less loaded hosts.
   • If other attempts to clear the alarm failed, please consider restarting the agent.
2. nsx-vdpi:
   • Gather information about DFW L7 rules (rules with a context profile attached).
   • The alarm relates to the high number of L7 flows needing inspections.
   • Please consider refining the L7 rule to restrict only L7 flows for inspections.
   • Please consider moving some workloads to less loaded hosts.
   • If other attempts to clear the alarm failed, please consider restarting the agent. Note: L7 inspections will be interrupted during agent restart.
   • Perform a restart of the agent:
3. Other agents: ESXi:
   • Login to the ESXi node as root
   • Restart agent {agent_name} by `/etc/init.d/{agent_name} restart`.

Maintenance window required for remediation? No