High Memory usage by Repmgr.exe on 4.0.x Windows Sensors

Products

Carbon Black Cloud Enterprise EDR (formerly Cb Threathunter)

Issue/Introduction

Endpoints with a high volume of activity may see an increase in memory utilization by the repmgr.exe process. This has most commonly been seen on Windows Servers, but can also be experienced on Windows desktops. The majority of endpoints do not experience this behavior. This issue is only experienced on endpoints with a high rate of activity and generally only in environments that have Enterprise EDR (EEDR) enabled.

The problem can be identified by:

Noting that repmgr.exe is consuming an excess of memory (5+ GB)
Opening an admin command prompt and running the commands:
```
"c:\program files\confer\repcli" counters | findstr /i AdminPort:KernelToUserBulkEventTransfer:SendDurationCount
```
- "Send DurationCount" indicates that the kernel sent X amount of messages
```
"c:\program files\confer\repcli" counters | findstr /i KernelComms:BulkEventTransfer(ms)Count
```
- BulkEventTransfer(ms)Count indicates that repmgr.exe processed X amount of messages

Having a delta between the first and second number indicates that repmgr.exe can't keep up with the rate of events coming from kernel. The larger the delta the more memory that will be utilized as any delta is stored in memory. The below example shows a delta of ~50K (4250241 - 4200346)

AdminPort:KernelToUserBulkEventTransfer:SendDurationCount=4250241
KernelComms:BulkEventTransfer(ms)Count=4200346

Environment

Carbon Black Cloud Windows Sensor: Versions 4.0.0, 4.0.1, and 4.0.2
Carbon Black Cloud Console: All versions
Microsoft Windows: All supported versions

Cause

This is issue UAV-3193 and CRE-19566
KernelToUserBulkEventTransfer message-processor queue under heavy load

Resolution

There are currently three options to workaround this:

Upgrade to sensor version 4.0.3
Utilize Event Reporting and Sensor Operation Exclusions to reduce the volume of activity the sensor needs to process.
- If you can identify the excessive noise on the system(s) in question this is the preferred solution.
Utilize the "PscrQueueMax" configuration

Event Reporting and Sensor Operation Exclusions (Preferred Solution)

Identify which applications are driving the most volume on the endpoints.
1. Open the Carbon Black Cloud console
2. Navigate to the Investigate page (Processes tab for customers that have both Endpoint Standard and Enterprise EDR)
3. Filter on the device experiencing the issue (ex device_id:111111)
4. Sort by Filemods, modloads, netcons, etc to identify the processes generating the most volume.
See Event Reporting and Sensor Operation Exclusions to configure an exclusion. For help contact Broadcom Carbon Black support.

PscrQueueMax (Not Preferred Solution)

This configuration can be set two different ways:

Manually

This method can be used when a select few endpoints in the environment are experiencing the behavior.

Per policy

This method can be used when there are a large quantity of endpoints experiencing the issue in the environment.
This requires Broadcom Carbon Black support for assistance

Manual Steps:

Enable bypass mode on the sensor from the Carbon Black Cloud Console
Get started with bypass mode
Open cfg.ini (C:\ProgramData\CarbonBlack\DataFiles\cfg.ini) with Notepad (Notepad++.exe with Admin privilege is recommended)
Add the following line:
```
PscrQueueMax=4000
```
Save changes to cfg.ini with "Save As" option; maintain the same file name and select a destination outside of the cfg.ini directory
Move the old cfg.ini file out of it's current directory (to keep as a backup)
Move the new cfg.ini file with the "PscrQueueMax" entry into C:\ProgramData\CarbonBlack\DataFiles\cfg.ini
Run the following repcli command. Review the following KB if needed: How to Access RepCLI Utility
```
"c:\program files\confer\repcli" updateconfig
```
Disable bypass mode on the sensor from the Carbon Black Cloud Console

Note: This configuration will persist until manually removed.

Per policy Steps:

Contact Broadcom Carbon Black Support. When requesting this, please provide the name of the policy the configuration should be set on. We recommend only setting this configuration for sensors that have experienced the issue.

Additional Information

An upcoming release will address this issue without the need for manual intervention