System Lockups and Performance Issues With Windows Agent 8.9 and Higher

Products

• Carbon Black App Control

Issue/Introduction

After installing any of the 8.9.x Agent versions (8.9.0, 8.9.2, 8.9.4, 8.9.6) Agent is installed or upgraded, the system begins to exhibit high CPU usage and application slowness that gets progressively worse until the system locks up.

Environment

• App Control Agent: 8.9.0 - 8.10.2
• Microsoft Windows: All Supported Versions

Cause

Two new features included with Windows Agent 8.9.0+ have caused stalls or additional analysis of file operations that contribute to Agent's performance overhead, especially on very busy systems.

Resolution

Disable Rule Expansion Timeout:

Rule Expansion Timeout was changed to be disabled by default with the release of Agent 8.10.0 (EPCB-21407). If on Agent version 8.9.0 - 8.9.6 add the following Agent Config:

1. Add the following Agent Config for All Windows Agents:
   • Name: Disable Rule Expansion Timeout (EPCB-21407)
   • Host ID: 0
   • Value: 

     kernelExpandRulesTimeoutMs=0

   • Platform: Windows
   • Create For: All Policies
2. Save and Exit
3. Verify Agent shows as Connected and Up to Date before testing again.

 

Disable Process Hollowing:

By default, Agents track the Process Hollowing operations, but unless the Process Hollowing Rapid Config is enabled this additional tracking can be disabled. A fix is scheduled to be included with Agent 8.10.2 that will properly toggle the Process Hollowing Detection based on the Rapid Config status (EPCB-21407).

1. Add the following Agent Config for All Windows Agents:
   • Name: Disable Process Hollowing Detection (EPCB-21764)
   • Host ID: 0
   • Value: 

     kernelDisableProcessHollowingDetection=1

   • Platform: Windows
   • Create For: All Policies
2. Save and Exit
3. Verify Agent shows as Connected and Up to Date before testing again.

If the issue persists, attempt a reboot to be sure the settings are fully applied. Follow the steps in Troubleshooting Agent Performance Issues.

Additional Information

• Agent 8.9.0 introduced Process Hollowing Detection. By default, the Agent tracks the Process Hollowing operations, but until the Rapid Config is enabled and configured, it will not take action. Engineering is investigating changing this default behavior to Disabled in a future release of the Agent (EPCB-21764).
• When a process or task "Runs as a different user" other than the currently logged in user (example: using Scheduled Tasks or Run As), then Agents re-expand all Custom Rules.
• kernelExpandRulesTimeoutMS (10sec) was added to handle rare circumstances when an immediate execution in this scenario would trigger Execution Blocks. This configuration instructs the Agent to stall some operations for a specific set of time to allow for Rule Expansion. This repeated stalling could cause performance issues, especially when large numbers of Custom Rules and running processes are present.
• Disabling Rule Expansion Timeout will not decrease the Security Posture of the Agent.
• Disabling Process Hollowing Detection will only impact the Security Posture of the Agent if the Rapid Config is also fully configured and enabled.