Troubleshooting External Event Logging (Syslog) Integration

Products

• Carbon Black App Control

Issue/Introduction

Steps to troubleshooting the External Event Logging (Syslog) settings.

Environment

• App Control Server: All Supported Versions
• External Event Logging

Resolution

1. Log in to the Console and navigate to Settings > System Configuration > Events.
   • Verify configuration settings are accurate for the external application.
2. Check with the Network Team to be sure UDP traffic on the specified port is allowed from the App Control Server to the SIEM.
   • Validate the App Control server can reach the address over the configured port using this Powershell command. 

     Test-NetConnection -ComputerName <ip or fqdn of syslog receiver> -Port 514

   • The App Control server can only forward to syslog through UDP. Validate the receiving server is configured to listen over UDP. 
   • Collect a Wireshark capture. Verify if data is being sent out of the server. 
     • Use this filter during the Wireshark capture along with the port being used.

       udp.port == 514

     • Click one of the  > Data >  Data. Validate there are "Carbon Black App Control" events being sent in the data packet. 
3. Verify the external application is configured to accept/ingest all events from the App Control Server.
   • Some applications only ingest events from a pre-defined list of machines.
   • Some applications can filter out events before displaying/fully processing them.
4. Run SQL Server Management Studio as the Carbon Black Service Account.
   1. Connect to the App Control database and execute the following query:

      USE das;
      SELECT 'Total Events in DB' [Database Info], CAST(FORMAT(COUNT(*), 'N0') AS NVARCHAR) [Value] FROM dbo.events (NOLOCK)
      UNION
      SELECT 'Oldest Event in DB' [Database Info], CAST(MIN(date_created) AS NVARCHAR) [Value] FROM dbo.events (NOLOCK)
      UNION
      SELECT 'Newest Event in DB' [Database Info], CAST(MAX(date_created) AS NVARCHAR) [Value] FROM dbo.events (NOLOCK)
      UNION
      SELECT 'Last Event ID in DB' [Database Info], CAST(FORMAT(MAX(event_id), 'N0') AS NVARCHAR) [Value] FROM dbo.events (NOLOCK);
      
      SELECT 'Last Event ID to Syslog' [Syslog Info], CAST(FORMAT(MAX(task_param), 'N0') AS NVARCHAR) [Value] FROM dbo.scheduled_tasks (NOLOCK) WHERE task = 'SyslogGetEvents'
      UNION
      SELECT 'Last Event ID Timestamp' [Syslog Info], CAST(date_created AS NVARCHAR) [Value] FROM dbo.events (NOLOCK) WHERE event_id = (SELECT task_param from dbo.scheduled_tasks (NOLOCK) WHERE task = 'SyslogGetEvents');

5. Review the results accordingly:
   • Last Event ID in DB and Last Event ID to Syslog should be similar. They will likely never match exactly, but there should not be a large (thousands or tens of thousands) gap.
   • Newest Event in DB and Last Event ID Timestamp should be similar. They likely will never match exactly, but there should not be a large (hours or days) gap.
6. If results are concerning, or the environment is nearing 20M Events in a 24 hour period
   • Follow the guidance in Reducing Events Generated By Agents to eliminate unneeded Events from being sent and processed by the Server.
   • Consider adjusting the Execution Group for SyslogGetEvents.
   • It's possible External Event Logging is having troubles keeping up with Incoming Events.

 

If the issue persists provide the following logs when opening a case with Support:

1. Consider the options currently available
2. Results of the SQL Query in step 4.
3. Result of Powershell command in step 2.
4. Screenshot of the External Event Logging configuration in the Console > Settings > System Configuration > Events.
5. Start a Wireshark capture on the application server hosting the Console.
   • Be sure the Port for the External Application is specified in Step 2.
6. For Linux based Syslog receiving servers,
   1. Collect a TCPdump. 

      tcpdump -vv -X -i any port 514

   2. Validate and provide the output the port is listening over UDP.

      netstat -pnltu | grep 514

7. Server High Debug Logs for 5-10 minutes during the Wireshark capture.
8. The following details
   • External Application Name (Splunk, Microsoft Sentinel, etc)
   • When the issue began (After upgrading the Server, changing hostname, etc)
   • Details of the issue (No events are showing up after initially configuring External Event Logging, etc)

Additional Information

• All Events are stored in the database (and exported) in UTC Time.
• App Control currently only supports sending Events via UDP.
• Once sent, Events are not confirmed to be received, or re-sent.