EDR: Many sensors going offline intermittently and nginx/sensor.access.log filled with 499 errors.

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Sensors intermittently going offline in the console.
• The "Live Query" option in general settings is enabled.
• Sensors' sensorcomm.log showing successful registers and submits but failing checkins as so:

2020-10-09 13:15:44  | https://<myserver>:443/sensor/register/37209                    | 0x00000000 | 0     | 33696      | 277      | 16       | 500                  | 0
2020-10-09 13:16:18  | https://<myserver>:443/sensor/checkin/37209                     | 0x80072ee2 | 12002 | 45303      | 173      | 0        | 500                  | 0
2020-10-09 13:17:44  | https://<myserver>:443/sensor/checkin/37209                     | 0x80072ee2 | 12002 | 44991      | 359      | 0        | 500                  | 0

• High CPU usage by the Redis and Sensorservices on the primary (master) server.
• The /var/log/cb/nginx/sensor.access.log is filled with 499, 502 and 504 failure codes
• The issue may have coincided with a large number of sensor upgrades or new installs of 7.x sensors.
• Live-response may also be slow and timeout

 

Environment

• EDR Server: 7.2.0 and 7.3.0

Cause

This is issue CB-33260.
At each check-in, sensorservices will check the Redis keys. When there are many keys and Live Query is enabled, this can slow down the call for check-ins

Resolution

Please disable Live Query if you are experiencing this issue. 

1. Log into the Console as an Admin
2. Select the name at the top right
3. Select Settings
4. Select Advanced Settings
5. Uncheck "Enable Live Query"

CB-33260 will be fixed in EDR server 7.4 .

Additional Information

• Live Query is currently in Beta
• Estimated fix for this issue will be in 7.4.0