Console Performance Issues and Growing Agent Backlog

Products

• Carbon Black App Control (formerly Cb Protection)

Issue/Introduction

• Console pages are slower than normal to load.
• Agent Backlog is growing.
• Historical ServerLog.bt9 includes "gave up" messages when inserting an Antibody into some table, example:

  (8332 ReportABList Thread 1)After 0 retries, gave up on re-running statement : SET NOCOUNT ON;{CALL dbo.InsertAntibody (?,?,?,?,?,?,?,?,?,?,?,?,?,?)}.

• High Debug ServerLog.bt9 includes messages similar to any of the following:

  (5244 ReportABList Thread 1) Statement returned error [102]: [Microsoft][SQL Server Native Client 11.0][SQL Server]Incorrect syntax near '288241'.
  Statement: INSERT INTO dbo.temp_antibody_instances VALUES

  (11824 ReportABList Thread 0) Statement returned error [102]: [Microsoft][SQL Server Native Client 11.0][SQL Server]Incorrect syntax near 'VALU'.
  Statement: INSERT INTO dbo.temp_antibody_instances VALU

  (812 ReportABList Thread 0)ABStorage::_LoadAntibodyFromDB - max size reached bufferSizeUsed = 2794, fileOpCount = 20

Environment

• App Control Server: Versions 8.9.0 - 8.10.x
• App Control Agent: Versions 8.0.x - 8.9.x

Cause

The Agent is reporting very large Yara Tags that exceed 1024 characters, causing SQL Statement truncation issues and errors during Server processing.

Resolution

A permanent fix was included in releases of both the Agent (8.10.2) and Server (8.11.0):

• EPCB-20744 (Server 8.11.0): Resolved an issue that could cause Agent Backlog to grow unexpectedly.
• EPCB-20745 (Agent 8.10.2): Adjusted logic to handling Yara Tags sent to Server.

 

In the meantime, temporarily adjust the Shepherd Config, MaxFileOpBufferMessages

1. Log in to the Console and navigate to: https://ServerAddress/shepherd_config.php
2. Select the Property: MaxFileOpBufferMessages
3. Change the Value to: 1
4. Restart the services:
   • Carbon Black App Control Server
   • Carbon Black App Control Reporter
5. Log back in to the Console test overall performance.

Additional Information

Important: This Shepherd Config should be reverted to the default Value of 20 after upgrading to Server 8.11.0+