Troubleshooting External Event Logging (Syslog) Integration

Products

• Carbon Black App Control

Issue/Introduction

Steps to troubleshooting the External Event Logging (Syslog) settings.

Environment

• App Control Server: All Supported Versions
• External Event Logging

Resolution

1. Log in to the Console and navigate to Settings > System Configuration > Events.
   • Verify configuration settings are accurate for the external application.
2. Check with the Network Team to be sure TCP traffic on the specified port is allowed from the App Control Server to the SIEM.
3. Verify the external application is configured to accept/ingest all events from the App Control Server.
   • Some applications only ingest events from a pre-defined list of machines.
   • Some applications can filter out events before displaying/fully processing them.
4. Run SQL Server Management Studio as the Carbon Black Service Account.
   1. Connect to the App Control database and execute the following query:

      USE das;
      SELECT 'Total Events in DB' [Database Info], CAST(FORMAT(COUNT(*), 'N0') AS NVARCHAR) [Value] FROM dbo.events (NOLOCK)
      UNION
      SELECT 'Oldest Event in DB' [Database Info], CAST(MIN(date_created) AS NVARCHAR) [Value] FROM dbo.events (NOLOCK)
      UNION
      SELECT 'Newest Event in DB' [Database Info], CAST(MAX(date_created) AS NVARCHAR) [Value] FROM dbo.events (NOLOCK)
      UNION
      SELECT 'Last Event ID in DB' [Database Info], CAST(FORMAT(MAX(event_id), 'N0') AS NVARCHAR) [Value] FROM dbo.events (NOLOCK);
      
      SELECT 'Last Event ID to Syslog' [Syslog Info], CAST(FORMAT(MAX(task_param), 'N0') AS NVARCHAR) [Value] FROM dbo.scheduled_tasks (NOLOCK) WHERE task = 'SyslogGetEvents'
      UNION
      SELECT 'Last Event ID Timestamp' [Syslog Info], CAST(date_created AS NVARCHAR) [Value] FROM dbo.events (NOLOCK) WHERE event_id = (SELECT task_param from dbo.scheduled_tasks (NOLOCK) WHERE task = 'SyslogGetEvents');

   2. If near 20M Events in a 24 hour period, it's possible the External Event Logging is having troubles keeping up with Incoming Events.
   3. The "Last Event ID in DB" should be near the "Last Event ID to Syslog".

 

If the issue persists provide the following logs when opening a case with Support:

1. Results of the SQL Query above.
2. Screenshot of the External Event Logging configuration in the Console > Settings > System Configuration > Events.
3. Start a Wireshark capture on the application server hosting the Console.
   • Be sure the TCP Port for the External Application is specified in Step 2.
4. Server High Debug Logs for 5-10 minutes during the Wireshark capture.
5. The following details
   • External Application Name (Splunk, Microsoft Sentinel, etc)
   • When the issue began (After upgrading the Server, changing hostname, etc)
   • Details of the issue (No events are showing up after initially configuring External Event Logging, etc)

Additional Information

• All Events are stored in the database (and exported) in UTC Time.
• App Control currently only supports sending Events via TCP.
• Once sent, Events are not confirmed to be received, or re-sent.