CB Response: WebUI's Online Sensor Count widely fluctuates during the day.

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• WebUI Administrator notices wide fluctuations / dips in online sensor count typically on a cluster with large number of minions and sensors.
• The key symptom is that the online sensor count (as seen in the server dashboard page in the UI) will drop below the expected value for a period of minutes to hours.
• There are failed checkin calls in the nginx access.log file on the master, with checkins failing with 50X errors.
• While Offline sensors appears to be a serious problem, the effected sensors continue to submit data to their minions and this data is available to all CB-related functionality.

 

Environment

CB Response Server: All versions prior to 6.3.0 

Cause

• There exists an inefficiency in the checkin code in sensorservices, related to the method for looking up sensor group certificates during checkin. The particular code path was causing excessive load on our datagrid component, causing checkins to take too long and timeout.

Resolution

This is fixed in CB-21170, which is on track to be included in 6.3.0 CB Response Server Version.

Additional Information

• The checkin code was changed so that it does not try to get the group certificates from the datagrid service on every checkin. The information will now be kept in memory by sensorservices and refreshed as necessary. 
• Live Response may be unavailable for a short time for effected sensors.