Sensor packet drop ratio is high

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

An alarm has been raised indicating that the Sensor is experiencing a high packet drop ratio. This means that the sensor is discarding a significant percentage of the network traffic it is supposed to be analyzing. This is a critical issue as it creates blind spots and means that threats within the dropped traffic will be missed.

Environment

vDefend SSP >= 5.1
NDR Sensor >= 5.1

Cause

Packet drops are a symptom of a system being overwhelmed. The sensor cannot process packets as fast as they are being received. The most common causes are:

1. System resource overload: If the sensor's CPU is consistently high or its available memory is exhausted, the system will not have the resources to pull packets from the network card's buffer in time, forcing the NIC to drop them.

2. Extreme traffic volume: The volume of traffic being sent to the sensor (e.g., from a SPAN or mirror port) exceeds the processing capacity of the Sensor. Even a healthy system has a finite limit on how many packets per second it can inspect.

 

Resolution

Please follow these troubleshooting steps from the NDR Sensor CLI to diagnose the cause of the packet drops.

1. Confirm and locate the packet drops: First, identify that the sensor that is dropping the packets from the alarm description. Refer metrics graph under System > NDR Sensor > NDR Sensors in the "Sensor Details" tab.

 

 

Locate the "Packets not processed" graph.

 

 

 

2. Check system resources: Packet drops are almost always a symptom of resource exhaustion. Refer to the specific knowledge base articles - "Sensor CPU usage is high" or "Sensor memory usage is high".

3. Review the traffic source: If the packet drops are consistently correlated with high traffic volume and high resource utilisation, it is likely the sensor is undersized for the network segment it is monitoring. The long-term solution is to either increase the sensor resources or refine the traffic being sent to it.

As an alternative to vertical scaling, consider a horizontal scaling strategy: deploying additional sensors and distributing the traffic load across multiple sensor instances.

If packet drop remains high after these steps and it is impacting the sensor's performance, then it is advisable to collect the NDR Sensor support bundle (refer documentation for how to collect support bundle) and raise a support ticket.