EDR: Sensors Stuck Sending Event Data to an Event-Less Master After Enabling Custom Sensor Certificate

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Sensors grow event and binary backlog because they are sending event data to the event-less Master node of a cluster
• C:\Windows\CarbonBlack\SensorComms.log shows the the wrong IP/FQDN for the minion node that the sensor is supposed to be sending events and binaries.

Environment

• EDR Server: 7.x and Higher
• EDR Windows Sensor: 6.x and Higher
• Microsoft Windows OS: All Supported Versions
• Custom certificate for sensor to server communication enabled

Cause

After enabling the custom sensor communication certificate setting, the following two registry keys were set with wrong IP/FQDN for the minion node they are supposed to report their data:

HKEY_LOCAL_MACHINE\SOFTWARE\CarbonBlack\config\SensorBackendServerName
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\carbonblackk\SensorBackendServerName

Resolution

1. On the affected endpoint, open the Registry Editor
2. Remove the following registry keys:

   HKEY_LOCAL_MACHINE\SOFTWARE\CarbonBlack\config\SensorBackendServerName
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\carbonblackk\SensorBackendServerName

3. Restart the Carbon Black Sensor service
4. Validate that event data is getting submitted successfully to the minion node
   1. Open a command prompt
   2. Run the following:

      sc control carbonblack 201

   3. Check the C:\Windows\CarbonBlack\Diagnostics\SensorComms.log for successful eventlog submits to the correct IP/FQDN of the assigned minion node.

Additional Information

• Examples of successful eventlog and binary(storefile) submits:

2020-03-06 18:32:39 | https://<minion node>:443/data/eventlog/reserve/2 | 0x00000000 | 0 | 16 | 0 | 0 | 500 | 0
2020-03-06 18:32:39 | https://<minion node>:443/data/eventlog/submit2/2 | 0x00000000 | 0 | 203 | 105888 | 0 | 500 | 509
2020-03-06 18:32:51 | https://<minion node>:443/data/storefile/check/2 | 0x00000000 | 0 | 31 | 82 | 72 | 500