Collect Agent Performance Logs

Products

• Carbon Black App Control (formerly Cb Protection)

Issue/Introduction

Steps to collect App Control Agent logs for performance cases.

Environment

• App Control Agent: All Supported Versions
• Microsoft Windows: All Supported Versions
• macOS: All Supported Versions
• Linux: All Supported Versions

Resolution

Step 1: Gather Relevant Background Information

• Date/Time performance issue started occurring (did any change precede the start of it?)
• Actions being performed when performance is degraded
• Is the performance associated with a specific application? (If yes, what is the application name?).
• Does the vendor of the application have a recommended exclusion list and has it been implemented?
• Any paths/processes known to be associated with the application
• Are there any blocks seen locally or within the App Control console during the performance issue?
• Are results the same if the Agent is stopped and unloaded?
  • Start and Stop The Agent Via Local Command Line / Terminal
• What is the time difference with Agent stopped & unloaded vs fully operational (enabled)?

 

Step 2: Gather the Logs

Windows

1. Open an administrative command prompt
2. Issue the commands:

   cd "C:\Program Files (x86)\Bit9\Parity Agent"
   dascli password GlobalCLIPassword
   dascli setconfigprop max_rolling_trace_size_mb=500
   dascli setconfigprop max_rolled_trace_logs_to_keep=5
   dascli resetcounters
   dascli flushlogs
   dascli tamperprotect 0
   dascli debuglevel 6
   dascli kerneltrace 4
   dascli nettrace 1
   dascli diagnostics +performance
   
   Important, if Agent 8.10.0+ also issue:
   dascli registerprotected 0

3. Start a Procmon capture.
4. Collect 5-10 mins of data during the ongoing performance issue.
5. Stop the Procmon capture and save "All Events" as a PML file.
6. In the administrative command prompt execute the following commands to capture and reduce the logging levels to normal:

   dascli capture "%userprofile%\Desktop\%computername%-Performance.zip"
   dascli password GlobalCLIPassword
   dascli setconfigprop max_rolling_trace_size_mb=50
   dascli setconfigprop max_rolled_trace_logs_to_keep=1
   dascli debuglevel 0
   dascli kerneltrace 2
   dascli nettrace 0
   dascli diagnostics -performance
   
   If Agent 8.10.0+ first issue:
   dascli registerprotected 1
   
   dascli tamperprotect 1

7. Upload all collected data.

macOS

1. Open Terminal
2. Issue the commands:

   cd /Applications/Bit9/Tools
   ./b9cli --password 'GlobalCLIPassword'
   ./b9cli --resetcounters
   ./b9cli --flushlogs
   ./b9cli --debuglevel 4
   ./b9cli --kerneltrace 4
   ./b9cli --nettrace 1

3. Open the macOS Activity Monitor (Applications > Utilities > Activity Monitor).
4. Click View > Update Frequency > Often (2 sec)
5. Collect 5-10 mins of data during the ongoing performance issue.
6. Return to Activity Monitor, and double click the process with the performance issue. Click Sample.
7. Once the Sample completes choose "Save" to export the results.
8. In Terminal, capture the Agent Logs and return logging to defaults:

   ./b9cli --capture ~/Desktop/`Hostname`-PerformanceLogs.zip
   ./b9cli --password 'GlobalCLIPassword'
   ./b9cli --debuglevel 0
   ./b9cli --kerneltrace 2
   ./b9cli --nettrace 0

9. Collect System Logs

   system_profiler -detailLevel full > ~/Desktop/sysinfo.txt
   tar -cvf ~/Desktop/`hostname`-DiagnosticReports.tar /Library/Logs/DiagnosticReports

10. Upload all collected data.

Linux

Proper investigation into Agent Performance issues will require recreating the issue while capturing specific logs. Failure to capture the REQUIRED logs as outlined below will delay the investigation and require new log captures. While RECOMMENDED, the snapshot of b9daemon and the FAPREDEP are not required unless specifically asked for by Engineering.

1. REQUIRED Increase the Agent's Debug Levels and capture 5-10 minutes of the performance issue
   1. Open Terminal and issue the following commands:

      cd /opt/bit9/bin
      ./b9cli --password GlobalCLIPassword
      ./b9cli --resetcounters
      ./b9cli --flushlogs
      ./b9cli --debuglevel 4
      ./b9cli --kerneltrace 4
      ./b9cli --nettrace 1
      

   2. Recreate performance issue for 5-10 minutes
   3. While the performance issue is present, capture the output from TOP

      top -b -c -w 512 -n 10 -d 5 >> /var/tmp/$HOSTNAME-top_output.txt
      

   4. Capture the logs & reset the Debug Levels
      

      sudo ./b9cli --capture /var/tmp/$HOSTNAME-PerformanceLogs.zip
      ./b9cli --password GlobalCLIPassword
      ./b9cli --debuglevel 0
      ./b9cli --kerneltrace 2
      ./b9cli --nettrace 0
      ./b9cli --flushlogs

2. REQUIRED Use cbp-linux-sys-info.sh (attached at bottom of article) to collect system info
   1. Download the attached script to the endpoint.
   2. Extract and execute the script.
      • If the endpoint is in a Medium or High Enforcement Policy the "cbp-linux-sys-info.sh" script will need to be granted an Approval prior to execution.
      • If the execution of the script gives you an invalid character output, you may need to clean the sh file by running the command below, then executing the script again:

        sed -i 's/\r$//' cbp-linux-sys-info.sh

   3. Once completed, collect the resulting "/tmp/cbp-linux-sys-info.tgz" file.
3. Recommended Snapshot b9daemon with gcore during the performance issue
   1. Verify gcore is present with the --version command, if missing install via Package Manager

      gcore --version
      sudo dnf install gdb

   2. Disable the Agent's Tamper Protection and capture a core of the b9daemon
      • The resulting gcore file will be very large and the system may feel hung for several minutes while the capture takes place.
      • Tamper Protection must first be disabled to allow gcore to properly pause the process and capture the memory.

      ./b9cli --tamperprotect 0
      sudo gcore -o /var/tmp/$HOSTNAME_core `pidof b9daemon`

4. Recommended Capture FAPREDEP while attempting to recreate the issue, and with the Agent is not running
   1. Stop & unload the Agent.
   2. Collect System Logs.

      sudo tar cvfz /var/tmp/$HOSTNAME-SystemLogs.tgz /var/log

   3. While the Agent is shutdown & unloaded, run the FAPREDEP script.
5. Upload all collected data.