Troubleshooting Agent/Server Backlog

Products

• Carbon Black App Control (formerly Cb Protection)

Issue/Introduction

Troubleshooting steps to take when Agent and/or Server Backlog is consistently above the Threshold.

Environment

• App Control Server: All Supported Versions

Cause

Agent/Server backlog increases when there is an influx of file activity such as Windows Patching, OS upgrades, or other software deployments.

Resolution

Important Reminders!  Some Backlog is expected, as Agents will always generate File and Event data for the Server to process. Temporary environmental changes (OS patches, new deployments, etc) could cause an influx that may take 48-72 hours to fully process in some instances. The steps below should be taken when Backlog is consistently above the Threshold in Settings > System Health > Backlog.

 

1. Use Tech Docs to verify the App Control Server is meeting:
   • OER: Server Documentation > relevant version > Server Operating Environment Requirements.
   • Disk Speed: Server Documentation > relevant version > SQL Server Configuration Guide > Validating Storage Performance 
   • Not meeting the OER (especially required disk thruput can cause unexpected backlog)
   • Common Performance Pitfalls
   • SQL Storage Requirements
   • Architecture by Endpoint Count
2. Consider AB Exclusions related to reducing processing of specific Events and/or Files to speed up processing
   • Server Backlog Caused by PowerShell activity
   • Server Backlog Caused by .NET Activity
   • Event Backlog Caused by Certificate Checked Events
   • Event Backlog Caused by Session Change Events
3. Reduce what the Agent sends back to the Server for processing
   • Consider discarding information about Locally Approved Microsoft Supporting Files at the Agent.
     • Microsoft files (ex: DLLs) may account for more than half (or more) of all of the files in the Windows environment.
     • This will not prevent the Agent from reporting on Block Events relating to Microsoft files.
   • Audit for Custom Rules/Rapid Configs that are potentially triggering or generating Events more frequently than necessary.
4. Reduce application server overhead where possible
   • Check for any Drift Reports (Reports > Baseline Drift > Reports) that are no longer needed, and Disable accordingly.
   • Check for any Event Rules (Rules > Event Rules) that are no longer needed, and Disable accordingly.
5. Note environmental specific details that could contribute to temporary influxes in Server Backlog
   • Operating System and other software patches
   • Mass deployment of software or other file changes
   • Large number of new Agent deployments
6. Upgrade to the latest Server version to eliminate performance enhancements and other Resolved Issues, examples:
   • Server 8.11.2: Resolved a performance issue in Yara tag processing that could affect the server backlog process.
   • Server 8.11.0: Server service runs as a 64-bit process now
   • Server 8.10.2: DailyPruneTask & Yara Tag improvements

If the issue still persists, open a Technical Support Case and provide all of the following:

• Screenshot of the current Server Backlog from Settings > System Health > Backlog
• Findings/changes from steps provided above
• Database Logs for Performance Issues (SQL Server)