Event backlog ingestion stalled due to overflow on InsertAllEventColumns in DAS database

Products

• Carbon Black App Control

Issue/Introduction

• New event ingestion is stalled due to an overflow on InsertAllEventColumns in DAS database
• Approval requests stop showing up in the App Control console
• Agent side backlog grows due to the ingestion stall
• Serverlog.bt9 shows error messages like this:

  [118653] <DATE> (11180 ReportEventList Thread 0)   Statement returned error [8115]: [Microsoft][SQL Server Native Client 11.0][SQL Server]Arithmetic overflow error converting IDENTITY to data type int.

Environment

• Carbon Black App Control Server: All Supported Versions
• Carbon Black App Control Agent: All Supported Versions
• Software Meter function enabled in App Control console

Cause

The DAS database table "dbo.metered_execs" was designed to use the INT (integer) value by default, which has a maximum value of 2,147,483,647. Once that limit is reached, the database will fail to ingest any more events that are related to that table.

Resolution

• WORKAROUND: Create a Support case with the results of the SQL Analysis Script collected from the performance troubleshooting KB and if acceptable the "dbo.metered_execs" table can be dropped and replaced with a new table that uses the BIGINT data type value that will not have the same limitation as the default table.
• FIX: Future versions of the App Control Server will address this issue and be designed to use the BIGINT data type and/or reuse the existing entries in the table to prevent overflow.

Additional Information

• This query can be used in SQL Management Studio on the DAS database to confirm if the INT data type limit has been hit:

  USE das;
  SELECT 
      OBJECT_NAME(object_id) AS TableName, 
      name AS ColumnName, 
      TYPE_NAME(system_type_id) AS DataType,
      CAST(last_value AS bigint) AS CurrentIdentityValue,
      2147483647 - CAST(last_value AS bigint) AS Remain
  FROM sys.identity_columns
  WHERE OBJECT_NAME(object_id) IN ('internal_events', 'metered_execs', 'events')
  ORDER BY Remain ASC;