Collect Sensor Performance Logs For the Linux Sensor

Products

• Carbon Black Cloud Enterprise EDR (formerly Cb Threathunter)
• Carbon Black Cloud Endpoint Standard (formerly Cb Defense)

Issue/Introduction

How To Collect Sensor performance logs on an Linux device running Carbon Black Cloud.

Environment

• Carbon Black Cloud Linux Sensor: All Supported Versions
• Linux: All Supported Versions

Resolution

Prerequisites:

• Ensure troubleshooting steps and information requests documented in How to Troubleshoot Sensor Performance Issues have been completed

Steps:

1. Determine the baseline for what is considered "typical utilization", for instance: "My application normally handles X transactions/second, with the sensor installed, it handles less transactions/second”

2. Ensure the utility "perf" is installed
3. Save the following code as a bash file, then run it from the affected machine as superuser, while the issue is present:

   #!/bin/bash
   
   DIRNAME=$(hostname)_cbc-perf-$(date +%Y-%m-%d_%H-%M-%S)
   mkdir $DIRNAME
   cd $DIRNAME
   
   echo "VMWare Carbon Black Cloud - Performance Metrics Gathering Script"
   echo "Working..."
   
   top -b -n 10 > cbc-kmod.top
   ps -efT > cbc-kmod.ps
   ps -efo uid,pid,pcpu,rss,spid,ppid,c,stime,tty,time,comm > cbc-cpu.ps
   
   #vmstat output 
   vmstat 3 10 > cbc.vmstat 
   
   #iostat output 
   iostat 3 10 > cbc.iostat
   
   #Backup kptr_restrict
   cp /proc/sys/kernel/kptr_restrict ./kptr_restrict_$(date +%Y-%m-%d_%H-%M-%S)
   
   # Get kptr_restrict value
   old_kptr_restrict=$(cat /proc/sys/kernel/kptr_restrict)
   
   # Disable kptr_restrict for a moment
   echo 0 > /proc/sys/kernel/kptr_restrict
   
   # Record everything for 15 seconds
   echo "Sleeping for 15ish seconds ..."
   perf record -o cbc-perf_sleep_15.data -F 99 -a -g sleep 15
   
   # Dump human readable data to file for easier reading
   perf report -i cbc-perf_sleep_15.data --hierarchy > cbc-perf_sleep_15.stdio
   
   # Copy kallsyms for kmod addresses
   cp /proc/kallsyms .
   
   cd ..
   
   # Restore kptr_restrict
   echo "${old_kptr_restrict}" > /proc/sys/kernel/kptr_restrict
   
   tar -zcf $DIRNAME.tgz $DIRNAME
   rm -rf $DIRNAME
   
   echo "Wrote $DIRNAME.tgz"
   echo "Please share $DIRNAME.tgz with support"

4. The script will output a compressed file with a .tgz extension
5. Collect Carbon Black Cloud Sensor Logs Locally
6. Bundle the two files and upload them to the case.

In cases where the cbagentd memory consumption is raising rapidly beyond 5GB+ within few minutes, perform below steps:

1. Stop and Start CBC Linux Sensor, and then enable below following debugs:
   

   [root@ log]# sudo systemctl stop cbagentd
   
   [root@ log]# sudo systemctl start cbagentd

   
   [root@ log]# /opt/carbonblack/psc/bin/repcli debug 1
   
   Sensor is in debug mode
   
   [root@ log]# /opt/carbonblack/psc/bin/repcli debug-th 1
   
   threat-hunter is in debug mode

2. Enable strace logging (The strace logging have to be running for a short period of time):

   strace -fkt -o strace.out -p `pgrep cbagentd` -e trace=mmap,munmap,brk

3.  Keep tracking the memory consumption, and if it hits the maximum or goes above 7GB, then stop the debugs and the strace logging (Ctrl+c), and collect debug logs:
   

   #For stopping the debugging of cbc sensor
   [root@ log]# /opt/carbonblack/psc/bin/repcli debug 0
   
   [root@ log]# /opt/carbonblack/psc/bin/repcli debug-th 0

4. Collect the stract.out, and the debug logs and attach it to your support ticket with Broadcom support.

Additional Information

• For CPU performance issues, confirm that the total usage is greater than 200% (2 CPU cores) sustained, because according to the Linux Sensor OER, the sensor is designed to use up to 2 full CPU cores during times of high event creation by default and still considered functional.