'sensor_comm_failures' Messages Filling /var/log/messages File

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

'sensor_comm_failures' messages are filling the /var/log/messages file. Example message:

sensor_comm_failures: {"sensor_timestamp": "2019-03-03T10:27:18.199000+00:00", "timestamp": "2019-03-03T10:27:39.402712+00:00", "sensor_id": 50016, "server_url": "https://12.345.67.89:443/data/storefile/check/50016", "failure_code": -2147014836}

Environment

• EDR Server: Version 6.2.4 and Higher

Cause

Logging of sensor communications failures has changed with the version 6.2.4 release. Prior to version 6.2.4 sensor communications failures were recorded in Postgres, with version 6.2.4 we are now writing them out to a log file. We make use of the rsyslog service running on the Linux system to handle the logging to this file. 

Resolution

1. On each node in a CB Cluster, open /etc/cb/cb.conf file to edit
2. Add following parameter/value:

CoreServicesRecordSensorDiagnostics=False

3. Restart CB Cluster services

Additional Information

• The additional logging with 'sensor_comm_failures' may cause delays with the 'sensorservices' process that handles sensor checkins, as it waits on rsyslog to accept additional messages before moving on.
• Adding the CoreServicesRecordSensorDiagnostics parameter does have the side effect that sensor diagnostics sent during sensor checkin are no longer being recorded, these include sensor comm failures and other sensor health metrics. It does not affect the core use of the product, but does limit visibility into any sensor related issues. Many of these metrics are however reported on the sensor details page. This change will persist through a restart as it is in the cb.conf. To enable the diagnostics data, simply remove that entry from cb.conf and restart services.