Cb Response: Syslog messages stop flowing to SIEM after recovery from network outage

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Network outage occurred between SIEM and Cb Response server
• Network was restored, but SIEM stopped recieving messages and did not resume after restore
• No output at all to /var/log/messages after network outage/restore

Environment

• Cb Response Server: all versions
• Syslog integration with SIEM

Cause

Rsyslog component of Linux OS may encounter an unrecoverable state during network outage when configured to send to external SIEM and will not automatically recover.

Resolution

1. Recover rsyslog by executing as root:

   service rsyslog restart 

2. Confirm messages are flowing: 

   tail -f /var/log/messages

Additional Information

When rsyslog encounters this type of condition, all logging will stop, including content to /var/log/messages and all other logs that use syslog.