Products

Carbon Black App Control

Issue/Introduction

Steps to collect specialized performance logs for the App Control Agent to help diagnose system performance degradation, such as high CPU or memory usage.

Environment

App Control Agent: All Supported Versions
Microsoft Windows: All Supported Versions
macOS: All Supported Versions
Linux: All Supported Versions

Resolution

Step 1: Gather Relevant Background Information

The following information will help during log analysis and provide valuable context into understanding the impact of the performance issue.

Date/Time performance issue started occurring (did any change precede the start of it?)
Actions being performed when performance is degraded
Is the performance associated with a specific application? (If yes, what is the application name?).
Does the vendor of the application have a recommended exclusion list and has it been implemented?
Any paths/processes known to be associated with the application
Are there any blocks seen locally or within the App Control console during the performance issue?
Are results the same if the Agent is stopped and unloaded?
What is the time difference with Agent stopped & unloaded vs fully operational (enabled)?

Step 2: Gather the Logs

Proper investigation into Agent Performance issues will require recreating the issue while capturing specific logs.
Failure to capture the REQUIRED logs outlined below will delay the investigation and require new log captures.
Recommended steps (ex: snapshotting b9daemon) are not required unless specifically requested.

Windows

REQUIRED Increase the Agent's Debug Levels via administrative command prompt

cd "C:\Program Files (x86)\Bit9\Parity Agent"
dascli password GlobalCLIPassword
dascli setconfigprop max_rolling_trace_size_mb=500
dascli setconfigprop max_rolled_trace_logs_to_keep=5
dascli resetcounters
dascli flushlogs
dascli tamperprotect 0
dascli debuglevel 6
dascli kerneltrace 4
dascli nettrace 1
dascli diagnostics +performance
dascli registerprotected 0

REQUIRED Start a Procmon capture.
Collect 5-10 mins of data during the ongoing performance issue.
Stop the Procmon capture, save "All Events" as a PML file, and zip the resulting PML file.

In the administrative command, capture the Agent logs then reduce the logging levels to default:

dascli capture "%userprofile%\Desktop\%computername%-Performance.zip"
dascli password GlobalCLIPassword
dascli setconfigprop max_rolling_trace_size_mb=50
dascli setconfigprop max_rolled_trace_logs_to_keep=1
dascli debuglevel 0
dascli kerneltrace 2
dascli nettrace 0
dascli diagnostics -performance
dascli registerprotected 1
dascli tamperprotect 1

Provide the compressed PML file and Agent Logs to Support.

macOS

REQUIRED Increase the Agent's Debug Levels and capture 5-10 minutes of the performance issue

Open Terminal and issue the commands:

cd /Applications/Bit9/Tools
./b9cli --password 'GlobalCLIPassword'
./b9cli --resetcounters
./b9cli --flushlogs
./b9cli --debuglevel 4
./b9cli --kerneltrace 4
./b9cli --nettrace 1

Recreate the performance issue for 5-10 minutes.

In Terminal, capture the Agent Logs and return logging to defaults:

./b9cli --capture ~HOME/Desktop/$(hostname)-PerformanceLogs.zip
./b9cli --password 'GlobalCLIPassword'
./b9cli --debuglevel 0
./b9cli --kerneltrace 2
./b9cli --nettrace 0

REQUIRED Use mac-sys-info.sh (attached at bottom of article) to collect system data, CPU and memory samples.
1. Download the attachment to the endpoint and execute the script
```
sudo sh appc-mac-sys-info.sh
```
  - If the endpoint is in Medium or High Enforcement, the script will need an Approval prior to execution.
  - If the execution of the script gives an invalid character, syntax or permission error, clean the file by running
```
sed -i '' 's/\r$//' ac-mac-sys-info.sh && chmod +x ac-mac-sys-info.sh
```
2. Once completed, collect the resulting file from /tmp/mac-sys-info-TIMESTAMP.tar.gz

Recommended Collect System Logs

system_profiler -detailLevel full > ~HOME/Desktop/sysinfo.txt
tar -cvf ~HOME/Desktop/$(hostname)-DiagnosticReports.tar /Library/Logs/DiagnosticReports

Upload all collected data.

Linux

REQUIRED Increase the Agent's Debug Levels and capture 5-10 minutes of the performance issue

Open Terminal and issue the following commands:

cd /opt/bit9/bin
./b9cli --password GlobalCLIPassword
./b9cli --resetcounters
./b9cli --flushlogs
./b9cli --debuglevel 4
./b9cli --kerneltrace 4
./b9cli --nettrace 1

Recreate performance issue for 5-10 minutes
While the performance issue is present, capture the output from TOP
```
top -b -c -w 512 -n 10 -d 5 >> /var/tmp/$HOSTNAME-top_output.txt
```

Capture the logs & reset the Debug Levels

sudo ./b9cli --capture /var/tmp/$HOSTNAME-PerformanceLogs.zip
./b9cli --password GlobalCLIPassword
./b9cli --debuglevel 0
./b9cli --kerneltrace 2
./b9cli --nettrace 0
./b9cli --flushlogs

REQUIRED Use cbp-linux-sys-info.sh (attached at bottom of article) to collect system info
1. Download the attached script to the endpoint.
2. Extract and execute the script.
  - If the endpoint is in a Medium or High Enforcement Policy the "cbp-linux-sys-info.sh" script will need to be granted an Approval prior to execution.
  - If the execution of the script gives you an invalid character output, you may need to clean the sh file by running the command below, then executing the script again:
```
sed -i 's/\r$//' cbp-linux-sys-info.sh
```
3. Once completed, collect the resulting "/tmp/cbp-linux-sys-info.tgz" file.
Recommended Snapshot b9daemon with gcore during the performance issue
1. Verify gcore is present with the --version command, if missing install via Package Manager
```
gcore --version
sudo dnf install gdb
```
2. Disable the Agent's Tamper Protection and capture a core of the b9daemon
  - The resulting gcore file will be very large and the system may feel hung for several minutes while the capture takes place.
  - Tamper Protection must first be disabled to allow gcore to properly pause the process and capture the memory.
```
./b9cli --tamperprotect 0
sudo gcore -o /var/tmp/$HOSTNAME_core `pidof b9daemon`
```
Recommended Capture FAPREDEP while attempting to recreate the issue, and with the Agent is not running
1. Stop & unload the Agent.
2. Collect System Logs.
```
sudo tar cvfz /var/tmp/$HOSTNAME-SystemLogs.tgz /var/log
```
3. While the Agent is shutdown & unloaded, run the FAPREDEP script.
Upload all collected data.

Attachments

ac-mac-sys-info.sh get_app

cbp-linux-sys-info.sh get_app

Collect Performance Logs for App Control Agent

Article ID: 286081

Updated On: