EDR: Server has large sensor backlog and nginx/access.log is flooded with HTTP 503 errors

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Large sensor backlog
• Performance degradation over time fixed by a system restart
• Network bandwidth usage degradation
• Thousands of HTTP 503 GET calls per second in /var/log/cb/nginx/access.log

::ffff:<IPADDR> - - [11/Aug/2017:16:09:36 -0400(0.026)] "GET /data/eventlog/reserve/2 HTTP/1.1" 503 334 "-" "" ">[::1]:9000" "-" "-" 60
::ffff:<IPADDR> - - [11/Aug/2017:16:09:36 -0400(0.026)] "GET /data/eventlog/reserve/2 HTTP/1.1" 503 334 "-" "" ">127.0.0.1:9000" "-" "-" 60
::ffff:<IPADDR> - - [11/Aug/2017:16:09:36 -0400(0.025)] "GET /data/eventlog/reserve/2 HTTP/1.1" 503 334 "-" "" ">[::1]:9000" "-" "-" 60

 

Environment

• EDR Server: 6.1.2 and above
• EDR Sensor: 6.0 - 6.1.1

Cause

This is a known issue with the older 6.x Windows sensors running on EDR Server versions above 6.1.2.

Resolution

1. Sensors running a 6.x version sensor below 6.1.2 must be upgraded
2. A temporary workaround while upgrades are rolled out can be applied to increase the wait time for request replies. These settings must be applied on each node of a cluster.
   • Edit /etc/cb/cb.conf
   • Add the following line 

     DatastoreSubmitTimeoutMs=1000

3. Restart EDR following the steps here

 

Additional Information

• This issue can occur even with a small number of early 6.x sensor versions. All affected sensors must be upgraded to resolve this issue. 
• If upgrading does not resolve the issue, additional research will be required to diagnose and address the initial cause of the performance degradation.