EDR: Event Forwarder failing with "too many open files" error

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• "Too many open files" error in /var/log/cb/integrations/cb-event-forwarder.log
• "Request payload size exceeds the limit" errors in /var/log/cb/integrations/cb-event-forwarder.log
• Event Forwarder stops processing and halts

Environment

• EDR Server: 6.x and Higher
• Event Forwarder (any version)

Cause

Failures to upload due to size limit on the SIEM side will be retried, eventually causing "Too many open files" error and halt in processing.

Resolution

1. Set bundle_size_max in /etc/cb/integrations/event-forwarder/cb-event-forwarder.conf to reflect the limit of your SIEM in bytes

bundle_size_max=10485760  //Default 10mb

2. Remove or backup any existing "event_bridge_output" files in output directory (defaults to /var/cb/data/) that are larger than the limit set
3. Restart Event Forwarder