Advisory: Event Submission Backlog Issue in Carbon Black EDR Windows Sensor 7.5.0

Products

• Carbon Black EDR

Issue/Introduction

Broadcom has identified a known issue affecting deployments running Carbon Black EDR Windows Sensor version 7.5.0 (7.5.0.19738) in clustered environments with eventless primary node. Under certain network conditions, the sensor may experience communication issues that can impact event submission and result in event backlogs on affected endpoints. This event submission issue is not recovered after the glitch is restored

In a clustered Carbon Black EDR deployment, sensors are designed to communicate with both Primary and Minion nodes according to established routing and load distribution mechanisms. However, a defect in Windows Sensor 7.5.0 can cause the sensor to incorrectly route traffic intended for a Minion node to the Primary node. This submission would be rejected if the primary node is configured to be eventless. The sensor will keep retrying submissions to the primary node only.

This behavior may occur following temporary infrastructure disruptions, including but not limited to:

• DNS resolution failures or delays
• Cluster node communication disruptions

Impact
Affected endpoints may experience one or more of the following symptoms:

• Check-In continues but no data is being ingested.
• HTTP 403 observed for data submit API calls in nginx access logs on primary node
• Growth of event backlog on the endpoint
• Delayed visibility of endpoint activity within the EDR console
• Potential increase in local sensor resource utilization while queued events await transmission

Importantly, endpoint monitoring and event collection continue locally on the sensor. However, event delivery to the Carbon Black EDR server may be delayed until the sensor  is restarted or the endpoint is rebooted.

Environment

• Carbon Black EDR Windows Sensor: 7.5.0 (7.5.0.19738)
• Windows: All Versions
• Clustered

Cause

Broadcom continues to investigate and address this defect and will provide additional updates as they become available.

Resolution

Broadcom is actively developing a broader and more permanent fix for this issue. The fix is planned for inclusion in Carbon Black EDR Windows Sensor 7.5.1, which is expected to become generally available (GA) in an upcoming release.

Broadcom recommends that all customers using 7.5.0 plan to upgrade to the 7.5.1 sensor release once it becomes available and has been validated within their environment.

Additional Information

• When upgrading to Carbon Black EDR Server version 7.9.2, review the following considerations carefully. These notes highlight key requirements, compatibility details, and important actions needed to ensure a smooth and successful upgrade process for Carbon Black EDR RPM Standalone and Cluster deployment

• At this time, no comprehensive workaround is available to permanently address the underlying defect. If symptoms are observed, restarting sensor services or rebooting the endpoint can temporarily relieve the issue (until it hits the issue again). Customers experiencing persistent issues should engage Broadcom Support for assistance with diagnosis and remediation guidance.

• Windows Sensor 7.5.0 Fails to Send Event Data to Minion Nodes Resulting in 403 Forbidden Errors