Collect Carbon Black Cloud Sensor Performance Logs
search cancel

Collect Carbon Black Cloud Sensor Performance Logs

book

Article ID: 292595

calendar_today

Updated On:

Products

Carbon Black Cloud Endpoint Standard (formerly Cb Defense) Carbon Black Cloud Enterprise EDR (formerly Cb Threathunter)

Issue/Introduction

This guide details the methods for gathering logs from a Carbon Black Cloud endpoint experiencing performance issues. It covers both collection via our Sensor Capture Script (Windows) and manually via Command Line Interface (CLI).

Environment

  • Carbon Black Cloud Sensor: All Supported Versions
  • Microsoft Windows: All supported versions
  • macOS: All Supported Versions
  • Linux: All Supported Versions

Resolution

Table of Contents

Windows

Option 1: Via Sensor Capture Script (Preferred)

Prerequisites

  1. RepCLI Authentication must be enabled (enabled during the initial sensor install  enabled on existing sensor installations ) AND you will need the deregistration code for the sensor available to enter when prompted.
  2. Create a folder where all logs will be saved. For the purposes of this document, this location will be referenced as C:\temp although the C:\temp file location can be replaced with whatever location you have specified for saving the log files. 
  3. Download the cbc-sensor-capture.ps1.zip attached to this article.
  4. Extract and copy cbc-sensor-capture.ps1 to C:\temp
  5. Download ProcmonLowAlt.exe.zip attached to the article How to Collect a low Altitude Procmon Capture.  Alternatively, follow the steps "Configure Procmon for Low Altitude" from the same article.
  6. Unzip procmon and copy to C:\temp
  7. Ensure wpr.exe exists in C:\Windows\System32\ 
NOTE: If C:\Windows\System32\wpr.exe does not exist, download Debugging Tools for Windows and at the "Select the features you want to download" install prompt deselect all other options except "Windows Performance Toolkit".  WPR.exe will download to C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit by default. Once downloaded copy wpr.exe to C:\Windows\System32\ 
 

Step 1: Capture Logs While The Sensor Is Active (Not in Bypass Mode)

In this section we want to reproduce the behavior when sensor is active. This consists of three separate sub-steps:

  • KeepEvents
    • This tells the sensor to retain the raw event files even after it's been sent to the backend. This is helpful as it helps us narrow down what the sensor was all seeing/doing at the exact time of the issue
  • Windows performance Recorder (WPR) Capture
    • This is a powerful Microsoft utility based on Event Tracing for Windows (ETW). The utility records system and application events.
  • Procmon Capture
    • Another Microsoft utility that shows system, registry, and process/thread activity.
    • This gives a different viewpoint than WPR. Combining information from WPR and Procmon can make it easier to diagnose the issue

KeepEvents

  1. Open Command Prompt using the "Run As Administrator" option
  2. Change Directory to C:\temp 
C:\WINDOWS\system32>cd to C:\temp
C:\temp>
  1. Run the following command to reset sensor counters and retain PSC events 
C:\temp>Powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 keepevents
  1. When the Enter the uninstall code to unlock restricted RepCLI commands.: prompt is presented, enter the uninstall code
  2. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the behavior
    1. Document the exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  3. Once behavior is fully reproduced, press Enter to exit the capture
  4. A zip will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
  5. Rename the .zip file by adding the prefix "keepevents-active" (i.e. keepevents-active-hostname-YYYYDDMMHHMMSS.zip)

Windows performance Recorder (WPR) Capture

  1. Run the following command to start wpr trace depending on the estimated time needed to reproduce
More than 5 minutes
C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 wpr
Less than 5 minutes 
C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 minifilter
  1. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the behavior
  2. Once behavior is fully reproduced, press Enter to exit the capture
    1. Document exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  3. A folder will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
    • NOTE: If the files are too large, the script may be unable to zip the folder. In this case, please manually compress the folder into a zip file.
    • NOTE: If the issue takes longer than 10 minutes to reproduce the issue, then the resulting log files may take up excessive disk space
  4. Rename the .zip file by adding the prefix "wpr-active" (i.e. wpr-active-hostname-YYYYDDMMHHMMSS.zip)

Procmon Capture

  1. Run the following command to start procmon
    C:\temp>powershell -executionpolicy bypass -f c:\temp\cbc-sensor-capture.ps1 procmon
  2. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the behavior
  3. Once behavior is fully reproduced, press Enter to exit the capture
  4. A zip will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
    • NOTE: If the files are too large, the script may be unable to zip the folder. In this case, please manually compress the folder into a zip file
    • NOTE: If the issue takes longer than 10 minutes to reproduce the issue, then the resulting log files may take up excessive disk space 
  5. Rename the .zip file by adding the prefix "procmon-active" (i.e. procmon-active-hostname-YYYYDDMMHHMMSS.zip)

Step 2: Capture Logs While The Sensor Is in Bypass Mode

In this section we want to reproduce the same exact steps when sensor is in bypass mode. This helps us compare what the system looks like when the sensor is no longer active and the machine is no longer experiencing the issue.

Windows performance Recorder (WPR) Capture

  1. Run the following command to start wpr trace depending on the estimated time needed to reproduce
More than 5 minutes
C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 wpr bypass
Less than 5 minutes 
C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 minifilter bypass
  1. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the issue
  2. Once issue fully reproduced, press Enter to exit the capture
    1. Document exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  3. A folder will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
  4. Rename the .zip file by adding the prefix "wpr-bypass" (i.e. wpr-bypass-hostname-YYYYDDMMHHMMSS.zip)

Procmon Capture

  1. Run the following command:
    C:\temp>powershell -executionpolicy bypass -f c:\temp\cbc-sensor-capture.ps1 procmon bypass
  2. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the issue
  3. Once the behavior has been reproduce, press Enter to exit the capture
  4. A folder will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
  5. Rename the .zip file by adding the prefix "procmon-bypass" (i.e. procmon-bypass-hostname-YYYYDDMMHHMMSS.zip)

Step 3: Zip all the logs

  1. Go to C:\temp and zip all the files that were created:
    • keepevents-active-hostname-YYYYDDMMHHMMSS.zip
    • wpr-active-hostname-YYYYDDMMHHMMSS.zip
    • procmon-active-hostname-YYYYDDMMHHMMSS.zip
    • wpr-bypass-hostname-YYYYDDMMHHMMSS.zip
    • procmon-bypass-hostname-YYYYDDMMHHMMSS.zip
      NOTE:       If an item from the above list is missing, re-review the steps above and collect it.
  2. Rename the zip as perfcapture-<date>-logs.zip 
  3. Upload the "perfcapture-<date>-logs.zip" file to the case for support to review

Option 2: Manually via CLI

Note: Only use this manual option if you cannot run the sensor_capture_script.ps1. Both methods achieve the same result, but the script is much faster and more efficient, so there is no need to complete both.

Prerequisites

  1. RepCLI Authentication must be enabled. If RepCLI Authentication was not enabled during the initial sensor install then RepCLI Authentication can be enabled on existing sensor installations
  2. Create a folder where all logs will be saved. For the purposes of this document, this location will be referenced as c:\temp although the c:\temp file location can be replaced with whatever location you have specified for saving the log files. 
  3. Procmon configured for a Low Altitude capture (may require reboot to setup).
  4. Windows Performance Recorder 
    • The command line version is included by default on Windows 10 and above.
    • The Graphical User Interface can be installed for other versions.

Reproduce the behavior when Sensor is Active

  1. Open a command line prompt using "Run as Administrator".
  2. Run the following command to start the WPR Trace (ETL) Log and reset counters depending on the estimated time needed to reproduce
Less than 5 Minutes 
C:\temp>wpr -start GeneralProfile -start CPU -start Registry -start FileIO -start DiskIO -start Minifilter -start Network -filemode
More than 5 Minutes 
C:\temp>wpr -start GeneralProfile -start CPU -start Registry -start FileIO -start DiskIO -start Network -filemode
  1. Reproduce the behavior
    1. Document exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  2. Collect the wpr-active.etl, counters.txt, and psc_sensor.zip 
A. C:\Program Files\Confer>repcli counters > C:\temp\counters.txt -- Change to desired location 
B. C:\temp>wpr -stop c:\temp\wpr-active.etl
C. C:\Program Files\Confer>repcli capture c:\temp -- Change to desired location
Collecting diagnostic data (this may take a few minutes)...
....
Captured diagnostic data in c:\temp\psc_sensor.zip
  1. Rename counters.txt to wpr-active-counters.txt, and psc_sensor.zip to wpr-active-psc_sensor.zip
  2. Run the following commands to collect the Procmon, Sensor logs, Counter logs :
A. C:\WINDOWS\system32>cd c:\program files\confer
B. C:\Program Files\Confer>sc qprotection cbdefense --Result will show either ANTIMALWARE LIGHT or None. If none skip to step H.
C. C:\Program Files\Confer>repcli bypass 1
D. C:\Program Files\Confer>repcli registerProtectedSvcs 0 
E. C:\Program Files\Confer>repcli stopCbServices
F. C:\Program Files\Confer>sc start cbdefense
G. C:\Program Files\Confer>repcli bypass 0
H. C:\Program Files\Confer>repcli deletepolicy 1DED7E47-CE4C-448E-AD01-6F4AC3CE7F5D
I. C:\Program Files\Confer>repcli resetcounters
  1. Launch Procmon.exe
  2. Start collection in Procmon (CTRL+E)
  3. Reproduce the behavior
    1. Document exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  4. Stop collection in Procmon (CTRL+E) and save the log file
    1. In the Save pop-up window, Select "Events to save: All events" and "Format: PML"
    2. Change the file name to procmon-active.PML
    3. Click "Save" and save to C:\temp 
  5. Run commands to collect counters.txt and psc_sensor.zip  
A. C:\Program Files\Confer>repcli counters > C:\temp\counters.txt -- Change to desired location
B. C:\Program Files\Confer>repcli capture c:\temp -- Change to desired location
Collecting diagnostic data (this may take a few minutes)...
....
Captured diagnostic data in c:\temp\psc_sensor.zip
  1. Rename counters.txt to procmon-active-counters.txt, and psc_sensor.zip to procmon-active-psc_sensor.zip


Reproduce the behavior when Sensor is in Bypass

  1. Place the sensor into bypass mode:
 C:\Program Files\Confer>repcli bypass 1
  1. Run the following command to start the WPR Trace (ETL) Log and reset counters depending on the estimated time needed to reproduce
Less than 5 Minutes 
C:\temp>wpr -start GeneralProfile -start CPU -start Registry -start FileIO -start DiskIO -start Minifilter -start Network -filemode
More than 5 Minutes 
wpr -start GeneralProfile -start CPU -start Registry -start FileIO -start DiskIO -start Network -filemode
  1. Reproduce the behavior
    1. Document exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  2. Collect the wpr-bypass.etl 
C:\temp>wpr -stop c:\temp\wpr-bypass.etl
  1. Launch Procmon.exe
  2. Start collection in Procmon (CTRL+E)
  3. Reproduce the behavior
    1. Document exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  4. Stop collection in Procmon (CTRL+E) and save the log file
    1. In the Save pop-up window, Select "Events to save: All events" and "Format: PML"
    2. Change the file name to procmon-bypass.PML
    3. Click "Save" and save to C:\temp
  5. Run commands to restore sensor:
A. C:\Program Files\Confer>repcli registerProtectedSvcs 1 -- Unnecessary if was skipped in step Sensor Active Step 6
B. C:\Program Files\Confer>repcli stopCbServices
NOTE: If using a sensor version prior to 3.6.0.1897, edit cfg.ini and remove "UnregisterProtected=True"
C. C:\Program Files\Confer>sc start cbdefense
D. C:\Program Files\Confer>repcli bypass 0
  1. Go to C:\temp, zip the files below and rename zip as perf-logs.zip
    1. wpr-active.etl
    2. wpr-active-counters.txt
    3. wpr-active-psc_sensor.zip
    4. wpr-bypass.etl
    5. procmon-active.PML
    6. procmon-active-counters.txt
    7. procmon-active-psc_sensor.zip
    8. procmon-bypass.PML
  2. Attach perf-logz.zip to your support ticket with Broadcom support.

macOS

  1. Collect Carbon Black Cloud Sensor Logs Locally
  2. Sysdiagnose results: Attempt to replicate the performance issue while executing Diagnose MacOS Network Issues Using Network Extension Diagnostics
  3. Replicate the issue while Activity Monitor is running
  4. Attach the log bundles to your support ticket with Broadcom support.

Linux

Note: For CPU performance issues, confirm that the total usage is greater than 200% (2 CPU cores) sustained, because according to the Linux Sensor OER, the sensor is designed to use up to 2 full CPU cores during times of high event creation by default and still considered functional.

  1. Determine the baseline for what is considered "typical utilization", for instance: "My application normally handles X transactions/second, with the sensor installed, it handles less transactions/second”

  2. Ensure the utility "perf" is installed
  3. Save the following code as a bash file, then run it from the affected machine as superuser, while the issue is present:
    #!/bin/bash

DIRNAME=$(hostname)_cbc-perf-$(date +%Y-%m-%d_%H-%M-%S)
mkdir $DIRNAME
cd $DIRNAME

echo "VMWare Carbon Black Cloud - Performance Metrics Gathering Script"
echo "Working..."

top -b -n 10 > cbc-kmod.top
ps -efT > cbc-kmod.ps
ps -efo uid,pid,pcpu,rss,spid,ppid,c,stime,tty,time,comm > cbc-cpu.ps

#vmstat output 
vmstat 3 10 > cbc.vmstat 

#iostat output 
iostat 3 10 > cbc.iostat

#Backup kptr_restrict
cp /proc/sys/kernel/kptr_restrict ./kptr_restrict_$(date +%Y-%m-%d_%H-%M-%S)

# Get kptr_restrict value
old_kptr_restrict=$(cat /proc/sys/kernel/kptr_restrict)

# Disable kptr_restrict for a moment
echo 0 > /proc/sys/kernel/kptr_restrict

# Record everything for 15 seconds
echo "Sleeping for 15ish seconds ..."
perf record -o cbc-perf_sleep_15.data -F 99 -a -g sleep 15

# Dump human readable data to file for easier reading
perf report -i cbc-perf_sleep_15.data --hierarchy > cbc-perf_sleep_15.stdio

# Copy kallsyms for kmod addresses
cp /proc/kallsyms .

cd ..

# Restore kptr_restrict
echo "${old_kptr_restrict}" > /proc/sys/kernel/kptr_restrict

tar -zcf $DIRNAME.tgz $DIRNAME
rm -rf $DIRNAME

echo "Wrote $DIRNAME.tgz"
echo "Please share $DIRNAME.tgz with support"
  4. The script will output a compressed file with a .tgz extension
  5. Collect Historical Carbon Black Cloud Sensor Logs
  6. Bundle the two files and upload them to the case.

In cases where the cbagentd memory consumption is raising rapidly beyond 5GB+ within few minutes, perform below steps:

  1. Stop and Start CBC Linux Sensor, and then enable below following debugs:
    [root@ log]# sudo systemctl stop cbagentd

    [root@ log]# sudo systemctl start cbagentd

    [root@ log]# /opt/carbonblack/psc/bin/repcli debug 1

    Sensor is in debug mode

    [root@ log]# /opt/carbonblack/psc/bin/repcli debug-th 1

    threat-hunter is in debug mode
  2. Enable strace logging (The strace logging have to be running for a short period of time):
    strace -fkt -o strace.out -p `pgrep cbagentd` -e trace=mmap,munmap,brk
  3.  Keep tracking the memory consumption, and if it hits the maximum or goes above 7GB, then stop the debugs and the strace logging (Ctrl+c), and collect debug logs:
    #For stopping the debugging of cbc sensor
    [root@ log]# /opt/carbonblack/psc/bin/repcli debug 0

    [root@ log]# /opt/carbonblack/psc/bin/repcli debug-th 0
  4. Collect the stract.out, and the debug logs and attach it to your support ticket with Broadcom support.

 

Additional Information

  • The ProcmonLowAlt.zip attached to How to Collect a low Altitude Procmon Capture was modified so that the configuration steps and reboot typically required if procmon is downloaded directly from Microsoft are not necessary; however, the modified version of procmon included in ProcmonLowAlt.zip has not been signed 
  • The WPR Trace cannot be collected at the same time as a Procmon Log
  • The PSC events should be collected separately from procmon and wpr if possible
  • Sensor Capture Script resets counters by default unless skipreset is specified

Attachments

cbc-sensor-capture.zip get_app
Powered by Wolken Software