EDR: CB-event-forwarder Not Sending Events Sometimes

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

Missing events from cb-event-forwarder, intermittently.

 Errors like:
 

"time="2019-02-12T14:47:11-05:00" level=error msg="Connection closed: Exception (501) Reason: \"write tcp XX.XX.XX.XX:port->XX.XX.XX.XX:5004: write: connection reset by peer\"" 
time="2019-02-12T14:51:29-05:00" level=info msg="Worker exiting" 
time="2019-02-12T14:51:30-05:00" level=info msg="Worker exiting" 
time="2019-02-12T14:47:11-05:00" level=info msg="Waiting for all workers to exit" 
time="2019-02-12T14:55:47-05:00" level=info msg="Worker exiting" 
time="2019-02-12T14:55:50-05:00" level=info msg="Worker exiting" 
time="2019-02-12T14:55:50-05:00" level=info msg="Worker exiting" 
time="2019-02-12T14:55:23-05:00" level=info msg="Worker exiting" 
time="2019-02-12T14:52:52-05:00" level=info msg="Worker exiting" 
time="2019-02-12T14:56:02-05:00" level=info msg="Worker exiting" 
time="2019-02-12T14:56:02-05:00" level=info msg="All workers have exited" 
time="2019-02-12T14:56:02-05:00" level=info msg="Rolling file /var/cb/data/event-forwarder/event-forwarder to /var/cb/data/event-forwarder/event-forwarder.2019-02-12T14:46:34.225.gz" 
time="2019-02-12T14:56:02-05:00" level=info msg="File handler configured to compress data" 
time="2019-02-12T14:56:02-05:00" level=info msg="AMQP loop 0 exited: Exception (501) Reason: \"write tcp XX.XX.XX.XX:port->XX.XX.XX.XX:5004: write: connection reset by peer\". Sleeping for 30 seconds then retrying." 
time="2019-02-12T14:56:14-05:00" level=info msg="Successfully uploaded file /var/cb/data/event-forwarder/event-forwarder.2019-02-12T14:46:34.225.gz to AWS S3.""

Environment

• EDR (Formerly CB Response) Server: All supported versions
• cb-event-forwarder: All supported versions

Cause

RabbitMQ is overloaded.

Resolution

Could be many reasons. Four possibilities here:

1) Too many event-forwarder subscriptions
Choose fewer subscriptions in /etc/cb/integrations/event-forwarder/cb-event-forwarder.conf

2) Too many event-forwarders connecting to one rabbitmq

3) Response server itself is busy

4) cb-event-forwarder post-processing option is enabled
Post-processing option can be disabled in in /etc/cb/integrations/event-forwarder/cb-event-forwarder.conf

Additional Information

The errors above show rabbit disconnecting. Forwarder waits for its workers. They exit. Forwarder rolls over the log file before it comes back up again. After 30 seconds from the initial error, the forwarder tries to publish again and succeeds. It took the forwarder more than the (default) 30 seconds to try again due to the number of workers in use and how long it took them to exit (looks like about the elapsed 5-8 minutes). This log is telling us that rabbitmq hung up on the forwarder. Rabbitmq is overloaded.