EDR Server: Event Ingest Rate Decreases After Server Restart

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Event and binary ingest rate decreases after server restart
• Growing backlog queue 
• Nginx access.log shows a growing number of 503s after restart
• Nginx submit requests show longer response times even if successful

Environment

• EDR Server: 7.6.x
• Site Throttling Enabled

Cause

Sensor site throttling was applied previously, but did not get applied until after restart

Resolution

• The actual behavior is expected with site throttling, the issue is that site throttling wasn't enabled sooner
• If the backlog is unacceptable, site throttling settings will need to be increased or disabled

Additional Information

• Site throttling reduces the amount of data accepted at a specified time. At those times, it would be expected that less sensor data would be ingested and backlog would increase
• 503 responses from the server are used to indicate that the sensor should retry sending data later. This is expected behavior if site throttle limits have been reached