High Memory usage by Repmgr.exe on 4.0.x Windows Sensors

Products

Carbon Black Cloud Endpoint Standard (formerly Cb Defense)

Issue/Introduction

Endpoints with a high volume of activity may see an increase in memory utilization by the repmgr.exe process. This has most commonly been seen on Windows Servers, but can also be experienced on Windows desktops. The majority of endpoints do not experience this behavior. This issue is only experienced on endpoints with a high rate of activity and generally only in environments that have Enterprise EDR (EEDR) enabled.

The problem can be identified by:

Noting that repmgr.exe is consuming an excess of memory (5+ GB)
Opening an admin command prompt and running the commands:
```
"c:\program files\confer\repcli" counters | findstr /i AdminPort:KernelToUserBulkEventTransfer:SendDurationCount
```
- "Send DurationCount" indicates that the kernel sent X amount of messages
```
"c:\program files\confer\repcli" counters | findstr /i KernelComms:BulkEventTransfer(ms)Count
```
- BulkEventTransfer(ms)Count indicates that repmgr.exe processed X amount of messages

Having a delta between the first and second number indicates that repmgr.exe can't keep up with the rate of events coming from kernel. The larger the delta the more memory that will be utilized as any delta is stored in memory. The below example shows a delta of ~50K (4250241 - 4200346)

AdminPort:KernelToUserBulkEventTransfer:SendDurationCount=4250241
KernelComms:BulkEventTransfer(ms)Count=4200346

Environment

Carbon Black Cloud Windows Sensor: Versions 4.0.0, 4.0.1, and 4.0.2
Carbon Black Cloud Console: All versions
Microsoft Windows: All supported versions

Cause

This is issue UAV-3193 and CRE-19566
KernelToUserBulkEventTransfer message-processor queue under heavy load

Resolution

There are currently three options to workaround this:

Downgrade to sensor version 3.9.2
Utilize the "PscrQueueMax" configuration
- This is only available for 4.0.0 and 4.0.1 sensors. This configuration can not be set on version 4.0.2.
Utilize Event Reporting and Sensor Operation Exclusions to reduce the volume of activity the sensor needs to process.
- If you can identify the excessive noise on the system(s) in question this is the preferred solution.
- This is the only option for endpoints that require the 4.0.2 sensor - such as those running Windows 11 24H2.

PscrQueueMax

This configuration can be set two different ways:

Manually

This method can be used when a select few endpoints in the environment are experiencing the behavior.

Per policy

This method can be used when there are a large quantity of endpoints experiencing the issue in the environment.
This requires Broadcom Carbon Black support for assistance

Manual Steps:

Enable bypass mode on the sensor from the Carbon Black Cloud Console
Get started with bypass mode
Open cfg.ini (C:\ProgramData\CarbonBlack\DataFiles\cfg.ini) with Notepad (Notepad++.exe with Admin privilege is recommended)
Add the following line:
```
PscrQueueMax=4000
```
Save changes to cfg.ini with "Save As" option; maintain the same file name and select a destination outside of the cfg.ini directory
Move the old cfg.ini file out of it's current directory (to keep as a backup)
Move the new cfg.ini file with the "PscrQueueMax" entry into C:\ProgramData\CarbonBlack\DataFiles\cfg.ini
Run the following repcli command. Review the following KB if needed: How to Access RepCLI Utility
```
"c:\program files\confer\repcli" updateconfig
```
Disable bypass mode on the sensor from the Carbon Black Cloud Console

Note: This configuration will persist until manually removed.

Per policy Steps:

Contact Broadcom Carbon Black Support. When requesting this, please provide the name of the policy the configuration should be set on. We recommend only setting this configuration for sensors that have experienced the issue.

Event Reporting and Sensor Operation Exclusions

See Event Reporting and Sensor Operation Exclusions. For help contact Broadcom Carbon Black support.

Additional Information

An upcoming release will address this issue without the need for manual intervention