Agent Memory Usage High is seen in the NSX-T UI Alarm

Products

• VMware NSX
• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Event ID: transport_node_health.agent_memory_usage_high
Alarm Description:

• Purpose:
  The Alarm for Agent Memory Usage indicates that some of your processes are with high memory usage/reservation. When the memory usage/reservation of this agent reaches the predefined high threshold, the alarm will be raised.
  Currently, the agents on the monitoring list include nsx-cfgagent, nsx-opsagent, nsx-nestdb, nsx-exporter and nsx-vdpi.
  • nsx-cfgagent - Handles the stateful configuration from controller.
  • nsx-opsagent - Handles all stateless operations initiated from NSX manager.
  • nsx-nestdb - Database storing all configurations and runtime states between controller and nsx-cfgagent.
  • nsx-exporter - Responsible for collecting statistics from transport node and sending it to the NSX manager.
  • nsx-vdpi - DFW L7 engine which does deep packet inspections.

Environment

VMware NSX

Resolution

Steps to Resolve

For 4.2.0 and higher

 

Recommended Action:

To avoid unexpected behavior in the monitored agent and process termination due to further increase in the memory usage you will need to try to reduce memory usage using the following procedure.

Note: Even after reducing workload or vMotioning VMs to less-loaded hosts, the memory reservation for the affected agent may remain elevated. The alarm will not clear unless the respective agent is restarted.

To confirm memory usage:

1. Run on ESXi as root user: nsxcli -c get firewall thresholds
2. If usage is within the limits, check the memory reservation values
   1. Run root user on ESXi: memstats -r group-stats -s name:min:max:consumed:conResv -u mb -g `vsish -e set /sched/groupPathNameToID host vim vmvisor vdpi`
   2. If conResv is high even though usage is low, the agent is holding onto elevated reservation values. This condition does not clear automatically.
   3. In this case, the agent needs to be restarted: /etc/init.d/{agent-name} restart
      
      

Agent Information:

1. nsx-cfagent:
   • Gather information about operations performed prior to observation of the alarm.
   • If the prior operation was adding more DFW rules and groups, please pause further additions until the alarm is cleared.
   • If workloads were vMotioned onto the host prior, please consider moving those workloads to less loaded hosts.
   • If the alarm persists, please consider moving some workloads to less loaded hosts.
   • If other attempts to clear the alarm failed, please consider restarting the agent.
2. nsx-vdpi:
   • Gather information about DFW L7 rules (rules with a context profile attached).
   • The alarm relates to the high number of L7 flows needing inspections.
   • Please consider refining the L7 rule to restrict only L7 flows for inspections.
   • Please consider moving some workloads to less loaded hosts.
   • If other attempts to clear the alarm failed, please consider restarting the agent. Note: L7 inspections will be interrupted during agent restart.
   • Perform a restart of the agent:
3. Other agents: ESXi:
   • Login to the ESXi node as root
   • Restart agent {agent_name} by `/etc/init.d/{agent_name} restart`.

Maintenance window required for remediation? No