Sensor memory usage is high

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

An alarm has been raised indicating that the Sensor's memory usage is consistently high. The alarm is triggered when the memory utilisation remains above 80% for more than 10 minutes. High memory utilisation can lead to system instability, dropped packets, and failure to analyse traffic, which can compromise security monitoring.

Environment

SSP 5.1

NDR Sensor 5.1

Cause

High memory usage on the sensor is typically caused by the intensive task of analysing network traffic. 

The most common cause is a sustained increase in the volume of network traffic that the sensor is configured to monitor.

Resolution

High memory usage should be temporary and may restore on its own. 

The long-term solution is to increase memory resources, refine the traffic being sent to the Sensor.

 

Please follow these troubleshooting steps from the NDR Sensor CLI using admin credentials to diagnose and resolve the issue:

1. Check overall memory statistics: Use the CLI command to get a detailed view of the system's memory usage.

ndr-sensor> get memory-stats

 

Pay close attention to these values:

MemTotal: The total physical RAM available.
MemFree: Memory that is completely unused.
MemAvailable: This is the most important metric. It's an estimate of how much memory is available for starting new applications, without swapping. If this value is critically low, the system is under memory pressure.

 

2. Increase memory resources: Sensor memory resources can be increased to handle high volume of traffic. Power off the sensor gracefully, increase the memory of the Sensor (default out-of-box configuration is 20 GB) through vSphere, and then power on the sensor again.

 

3. Check traffic metrics: The volume of traffic can be checked on the SSP UI. Refer metrics graph under System > NDR Sensor > NDR Sensors in the "Sensor Details" tab.

 

Now check the "Traffic Trends" - "Received traffic" and "Packets not processed"

If the volume of received traffic or packets dropped are too high, then investigate the network sources sending traffic to the Sensor's sniffing interfaces. Refine or/and reduce the volume of traffic being sent to the sniffing interfaces.

 

4. Turn off unused features: If you don't need a Sensor's features, turn off the features' data collection in the System > Data Collection tab of SSP.

 

5. Identify the high memory usage process: Use the process monitor command to get a real-time view of the processes that are consuming the most memory.

ndr-sensor> get process monitor

 

Observe the list for a few moments and identify which process is consuming most % of memory. Report this information incase a support ticket has to be opened

As an alternative to vertical scaling, consider a horizontal scaling strategy: deploying additional sensors and distributing the traffic load across multiple sensor instances.

 

Note: If memory usage remains high after these steps and it is impacting the sensor's performance, then it is advisable to collect the NDR Sensor support bundle (refer documentation for how to collect support bundle) and raise a support ticket.