Applications Hang or Fail to Start with Agent 8.9.2

Products

• Carbon Black App Control (formerly Cb Protection)

Issue/Introduction

• After upgrading to Agent 8.9.2 some applications are encountering hung processes or other performance issues
• High CPU usage by the Parity process
• Systems that start lots of processes Ex: build machines, batch jobs, automation/orchestration servers, RDS servers are most likely to be affected
• To confirm Agents are encountering the issue:
  1. Log in to the Console and navigate to: https://ServerAddress/Events.php?InternalEvents
  2. Change the Max Age accordingly
  3. Show Filters and add the following filters:

     Agent Version > Contains: 8.9.2
     Description > Contains: pParentProcess->IsForeRunnerOf

  4. If any events are returned then the Agents are affected

Environment

• App Control Agent: 8.9.2
• Microsoft Windows: All Supported Versions

Cause

Additional kernel logging was added in version 8.9.2 to make it easier to solve rarely reproducible issues. However, that additional logging caused a deadlock on systems that generate large numbers of processes.

Resolution

This issue was tracked under EP-20098 and resolved with the release of Agent version 8.9.4. Upgrading will resolve the issue.

Additional Information

• Please open a Support case to obtain instructions to adjust these settings in bulk for Agents encountering this issue.
• By default the Kernel Tracing level is 2, even when the Agent is in a Disabled Policy.
• Agents will continue to generate Internal Events matching pParentProcess->IsForeRunnerOf even with Kernel Tracing set to 0. 
• Setting the Kernel Tracing to 0 prevents the situation that causes the Agent to hold on to the process threads.
• Previously the work around was:
  1. Find the relevant Host ID for the impacted machine(s)
  2. Log in to the application server hosting the Console as the Carbon Black Service Account
  3. Launch SQL Server Management Studio and execute the following query:

     use das;
     UPDATE hostmain SET kernel_debug_level=0, debug_level=0, debug_flags=debug_flags|2, refresh_flags=refresh_flags|4
     WHERE host_id = <Host_ID>
     EXEC dbo.ExpireHostSession <Host_ID>

  4. Reboot the endpoint(s) in question when convenient to terminate the existing hung processes
  5. After upgrading to an Agent version 8.9.4+ please reset the Kernel Trace to default with this SQL script:

     use das;
     UPDATE hostmain SET kernel_debug_level=2, debug_level=0, debug_flags=debug_flags|2, refresh_flags=refresh_flags|4
     WHERE host_id = <Host_ID>
     EXEC dbo.ExpireHostSession <Host_ID>