EDR: Backlog grows after upgrading sensors to 6.2.1

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Growing backlog
• Server Nginx logs more non-200 HTTP results than 200's

Environment

• EDR Server: All Versions
• EDR Sensor: 6.2.1
• Microsoft Windows: All Supported Versions

Cause

Sensor is ignoring throttling when receiving an unexpected HTTP status code while the server is highly stressed.

Resolution

1. If the symptoms are being experienced, downgrade to 6.1.9 or upgrade to 6.2.2 or higher to resolve
2. If no symptoms are seen, sensors can stay on 6.2.1
3. A temporary workaround can be applied to delay sensor check-ins. The settings must be applied on each node of a cluster.
   1. Edit /etc/cb/cb.conf
   2. Add the following line 

      • DatastoreSubmitTimeoutMs=1000

   3. Restart EDR Server 

Additional Information

• CB-25763
• Currently this is linked to EDR Servers in highly stressed situations believed to be sizing issues.
• The EDR Sensor receives an unexpected HTTP status code from the EDR Server and in rapid succession tries to communicate
• This can cause potential instability on the server in the form of a DDOS if a sufficient number of sensors become similarly impacted.
• This instability in the form of a DDOS can also be seen with the 6.0.2 EDR Sensor for Windows; any 6.0.2 Windows sensors should be upgraded to an unaffected sensor version.