Collecting Sensor Performance Logs with Sensor Capture Script
search cancel

Collecting Sensor Performance Logs with Sensor Capture Script

book

Article ID: 284850

calendar_today

Updated On:

Products

Carbon Black Cloud Endpoint Standard (formerly Cb Defense) Carbon Black Cloud Enterprise EDR (formerly Cb Threathunter)

Issue/Introduction

Steps to collect Process Monitor (Procmon) Logs, Windows Performance Recorder (WPR) ETL Trace, and Sensor Diagnostic logs with Sensor Capture Script for troubleshooting issues sensor performance issues

Environment

  • Carbon Black Cloud Sensor: All Supported Versions
  • Windows: All Supported Versions

Resolution

Prerequisites

  1. RepCLI Authentication must be enabled. If RepCLI Authentication was not enabled during the initial sensor install then RepCLI Authentication can be enabled on existing sensor installations
  2. Create a folder where all logs will be saved. For the purposes of this document, this location will be referenced as C:\temp although the C:\temp file location can be replaced with whatever location you have specified for saving the log files. 
  3. Download the cbc-sensor-capture.ps1.zip attached to the Support Case
  4. Copy cbc-sensor-capture.ps1 to C:\temp
  5. Download ProcmonLowAlt.exe.zip attached to the article How to Collect a low Altitude Procmon Capture.  Alternatively, follow the steps "Configure Procmon for Low Altitude" from the same article.
  6. Unzip procmon and copy to C:\temp
  7. Ensure wpr.exe exists in C:\Windows\System32\ 
NOTE: If C:\Windows\System32\wpr.exe does not exist, download Debugging Tools for Windows and at the "Select the features you want to download" install prompt deselect all other options except "Windows Performance Toolkit".  WPR.exe will download to C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit by default. Once downloaded copy wpr.exe to C:\Windows\System32\ 
 

Step 1: Capture Logs While The Sensor Is Active (Not in Bypass Mode)

In this section we want to reproduce the behavior when sensor is active. This consists of three separate sub-steps:

  • KeepEvents
    • This tells the sensor to retain the raw event files even after it's been sent to the backend. This is helpful as it helps us narrow down what the sensor was all seeing/doing at the exact time of the issue
  • Windows performance Recorder (WPR) Capture
    • This is a powerful Microsoft utility based on Event Tracing for Windows (ETW). The utility records system and application events.
  • Procmon Capture
    • Another Microsoft utility that shows system, registry, and process/thread activity.
    • This gives a different viewpoint than WPR. Combining information from WPR and Procmon can make it easier to diagnose the issue

KeepEvents

  1. Open Command Prompt using the "Run As Administrator" option
  2. Change Directory to C:\temp 
C:\WINDOWS\system32>cd to C:\temp
C:\temp>
  1. Run the following command to reset sensor counters and retain PSC events 
C:\temp>Powershell -ExecutionPolicy bypass -f cbc-sensor-capture.ps1 keepevents
  1. When the Enter the uninstall code to unlock restricted RepCLI commands.: prompt is presented, enter the uninstall code
  2. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the behavior
    1. Document the exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  3. Once behavior is fully reproduced, press Enter to exit the capture
  4. A zip will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
  5. Rename the .zip file by adding the prefix "keepevents-active" (i.e. keepevents-active-hostname-YYYYDDMMHHMMSS.zip)

Windows performance Recorder (WPR) Capture

  1. Run the following command to start wpr trace depending on the estimated time needed to reproduce
More than 5 minutes
C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 wpr
Less than 5 minutes 
C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 minifilter
  1. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the behavior
  2. Once behavior is fully reproduced, press Enter to exit the capture
    1. Document exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  3. A folder will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
    • NOTE: If the files are too large, the script may be unable to zip the folder. In this case, please manually compress the folder into a zip file.
    • NOTE: If the issue takes longer than 10 minutes to reproduce the issue, then the resulting log files may take up excessive disk space
  4. Rename the .zip file by adding the prefix "wpr-active" (i.e. wpr-active-hostname-YYYYDDMMHHMMSS.zip)

Procmon Capture

  1. Run the following command to start procmon 
    C:\temp>powershell -executionpolicy bypass -f c:\temp\cbc-sensor-capture.ps1 procmon
  2. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the behavior
  3. Once behavior is fully reproduced, press Enter to exit the capture
  4. A zip will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
    • NOTE: If the files are too large, the script may be unable to zip the folder. In this case, please manually compress the folder into a zip file
    • NOTE: If the issue takes longer than 10 minutes to reproduce the issue, then the resulting log files may take up excessive disk space 
  5. Rename the .zip file by adding the prefix "procmon-active" (i.e. procmon-active-hostname-YYYYDDMMHHMMSS.zip)

Step 2: Capture Logs While The Sensor Is in Bypass Mode

In this section we want to reproduce the same exact steps when sensor is in bypass mode. This helps us compare what the system looks like when the sensor is no longer active and the machine is no longer experiencing the issue.

Windows performance Recorder (WPR) Capture

  1. Run the following command to start wpr trace depending on the estimated time needed to reproduce
More than 5 minutes
C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 wpr bypass
Less than 5 minutes 
C:\temp>powershell -ExecutionPolicy bypass -f c:\temp\cbc-sensor-capture.ps1 minifilter bypass
  1. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the issue
  2. Once issue fully reproduced, press Enter to exit the capture
    1. Document exact steps taken to reproduce to provide to Support
    2. Note local machine time when testing started, and timezone of machine
    3. Note local time when issue occurred
  3. A folder will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
  4. Rename the .zip file by adding the prefix "wpr-bypass" (i.e. wpr-bypass-hostname-YYYYDDMMHHMMSS.zip)

Procmon Capture

  1. Run the following command: 
    C:\temp>powershell -executionpolicy bypass -f c:\temp\cbc-sensor-capture.ps1 procmon bypass
  2. When "Hit enter to stop trace and collect logs.:" presents, start to reproduce the issue
  3. Once the behavior has been reproduce, press Enter to exit the capture
  4. A folder will be created with the device hostname, date and time of the capture (i.e. hostname-YYYYDDMMHHMMSS.zip).
  5. Rename the .zip file by adding the prefix "procmon-bypass" (i.e. procmon-bypass-hostname-YYYYDDMMHHMMSS.zip)

Step 3: Zip all the logs

  1. Go to C:\temp and zip all the files that were created:
    • keepevents-active-hostname-YYYYDDMMHHMMSS.zip
    • wpr-active-hostname-YYYYDDMMHHMMSS.zip
    • procmon-active-hostname-YYYYDDMMHHMMSS.zip
    • wpr-bypass-hostname-YYYYDDMMHHMMSS.zip
    • procmon-bypass-hostname-YYYYDDMMHHMMSS.zip
      NOTE:       If an item from the above list is missing, re-review the steps above and collect it.
  2. Rename the zip as perfcapture-<date>-logs.zip 
  3. Upload the "perfcapture-<date>-logs.zip" file to the case for support to review

Additional Information

  • The ProcmonLowAlt.zip attached to How to Collect a low Altitude Procmon Capture was modified so that the configuration steps and reboot typically required if procmon is downloaded directly from Microsoft are not necessary; however, the modified version of procmon included in ProcmonLowAlt.zip has not been signed 
  • The WPR Trace cannot be collected at the same time as a Procmon Log
  • Sensor Capture Script resets counters by default unless skipreset is specified
  • The keepevents parameter requires the uninstall code even if "Require code to uninstall sensor" is not enabled on the sensor policy
  • The PSC events should be collected separately from procmon and wpr if possible
Powered by Wolken Software