EDR Server: Linux system out-of-memory errors and process table contains 10's of thousands processes

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

1. Searching /var/log/messages for "memory" shows many out of memory type errors.
2. General slowness and connectivity issues to and from the server / cluster nodes
3. Listing the process table (ps -ef) shows many thousands of system-related processes like:

root     1940374 1940317 10:16 00:00:00  1236 105344 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
root     1952249 1952231 Nov28 00:00:00  1236 105344 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
root     1956219 1956200 Nov28 00:00:00  1236 105344 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root

root      536767    2145 Nov21 00:00:00   504  57508 /usr/sbin/CROND -n
root      538263    2145 Nov11 00:00:00   504  57508 /usr/sbin/CROND -n
root      539343    2145 Nov27 00:00:00   504  57508 /usr/sbin/CROND -n

root     3149802 3149801 Nov19 00:00:00  5128  24980 find -L /var/run/ -name *syslogd.pid
root     3152103 3152102 Nov21 00:00:00  5128  24980 find -L /var/run/ -name *syslogd.pid
root     3162402 3162401 Nov21 00:00:00  5128  24980 find -L /var/run/ -name *syslogd.pid

Environment

• EDR Server:  All versions
• EDR Sensor:  All versions
• Linux OS: All versions

Cause

The Linux host OS may be experiencing NFS mounting issues. 

Resolution

1. fix the NFS issues
2. reboot the Linux servers to clean up memory and the process table
3. Start the EDR Server/Cluster.