Sensor CPU Usage is High.

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

An alarm has been raised indicating that the Sensor's CPU usage has been consistently high. The alarm is triggered when the CPU utilisation remains above 80% for more than 10 minutes. This can lead to performance degradation, packet loss, and potentially missed threats.

Environment

SSP 5.1

NDR Sensor 5.1

Cause

High CPU usage on the sensor is typically caused by the intensive task of analysing network traffic. 

The most common cause is a sustained increase in the volume of network traffic that the sensor is configured to monitor. The more traffic, the more work the analysis engines must do.

Resolution

High CPU usage should be temporary and may restore on its own.

Please follow these troubleshooting steps to diagnose and resolve the issue:

The long-term solution is to increase CPU resources, refine the traffic being sent to the Sensor or deploy more Sensors.

1. Increase CPU resources: Sensor CPU resources can be increased to handle high volume of network traffic. Power off the sensor gracefully, increase the number of CPUs of the Sensor (default out-of-box configuration has 12 vCPUs) through vSphere, and then power on the sensor again.

 

2. Check traffic metrics: The volume of traffic can be checked on the SSP UI. Refer metrics graph under System > NDR Sensor > NDR Sensors in the "Sensor Details" tab.

 

Now check the "Traffic Trends" - "Received traffic" and "Packets not processed"

 

If the volume of received traffic or packets dropped are too high, then investigate the network sources sending traffic to the sensor's sniffing interfaces. Refine or/and reduce the volume of traffic being sent to the sniffing interfaces.

 

3. Turn off unused features: If you don't need a Sensor's features, turn off its data collection in the System -> Data Collection tab of SSP.

 

4. Identify the High CPU process: Use the process monitor command to get a real-time view of the processes that are consuming the most CPU. 

Please follow these troubleshooting steps from the NDR Sensor CLI using admin credentials:

ndr-sensor> get process monitor

Press "Shift + P" to sort the processes by CPU usage. Observe the list for a few moments and identify which process is consistently at the top. Report this information incase a support ticket has to be opened.

As an alternative to vertical scaling, consider a horizontal scaling strategy: deploying additional sensors and distributing the traffic load across multiple sensor instances.

 

Note: If CPU usage remains high after these steps and it is impacting the sensor's performance, then it is advisable to collect the NDR Sensor support bundle (refer documentation for how to collect support bundle) and raise a support ticket.