High CPU usage from Linux Sensor version 7.1.2

Products

• Carbon Black EDR (formerly Cb Response)
• Carbon Black EDR

Issue/Introduction

Endpoints running Linux EDR Sensor 7.1.2 will have spikes of high CPU usage, followed by sudden drops in the utilization. The high CPU spikes can cause system freezes and crashes resulting in dump files.

Environment

• EDR Sensor: Version 7.1.2
• Linux OS: All Supported Versions

Cause

Sensor is not able to handle  event rate burst scenarios where the Linux sensor starts seeing raw events getting dropped in the perf buffers. This leads to the sensor missing exit events leading to stale entries in the process tracking table.

Resolution

This behavior is going to be fixed in the 7.3.1 EDR Linux sensor version that is not yet released.

Additional Information

Workaround options:

• Decrease the kernel configuration variable kernel.pid_max to be 32K instead of 4M and reboot. This will hopefully limit the number of possible entries in the process tracking table and thereby limit memory usage.

  1. Add an entry to /etc/sysctl.conf like "kernel.pid_max=32768", 
  2. Reboot OS.

• Add a hardcoded memory usage limit to the sensor systemd unit file.

  1. Edit /etc/systemd/system/cbdaemon.service and add entry like "MemoryMax=1G" to the [Service] section
  2. Execute "systemctl daemon-reload"
  3. Execute "systemctl restart cbdaemon"
  4. Verify that this worked by running "systemctl status cbdaemon" and checking for the presence of a memory limit in the output.