EDR: Empty diagnostic and syslog notifications logs

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Empty diagnostic log files under /var/log/cb/ directories
• An rsyslog configuration file, like /etc/rsyslog.conf may be pointing to an invalid server.
• SIEM is not longer receiving events
• All sensors may appear as "offline" in the Web UI sensors page
• This error appears in /var/log/cb/sensorservices/startup.log:

  ERROR] Socket error processing request.

Environment

• EDR Server: All Versions
• Rsyslog
• SIEM

Cause

A configuration file is setup over TCP with a remote host that is not responding. Rsyslog is unable to continue it's processing and triggers an indefinite hang.

 

Resolution

1. Issue can be confirmed using the following command responding with (SYN_SENT)

   [root@CBResponse /]# lsof -c rsyslogd -ai
   
   COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
   rsyslogd 8376 root   17u  IPv4  66059      0t0  TCP SIEM:45514->10.160.9.8:shell (SYN_SENT)

2. Check that the remote host is reachable and the address is correct
3. Restart the rsyslog services to kick it back after the correction has been made

   CentOS 6: sudo service rsyslog restart
   CentOS 7: sudo systemctl restart rsyslog

Additional Information

• Log files affected (Note: There may be more system files)

/var/log/cb/job-runner/*
/var/log/cb/coreservices/*
/var/log/cb/sensorservices/*
/var/log/cb/enterprise/*
/var/log/cb/audit/*
/var/log/cb/live-response/*
/var/log/cb/notifications/*
/var/log/messages

• It's recommended to configure ActionQueue to help when host is unreachable

$WorkDirectory /var/lib/rsyslog # where to place spool files 
$ActionQueueFileName queue77 # unique name prefix for spool files 
$ActionQueueMaxDiskSpace 1g # 1gb space limit (could also use "50m") 
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown 
$ActionQueueType LinkedList # run asynchronously 
$ActionResumeRetryCount -1 # infinite retries if host is down

• ActionQueue settings should be handled by an Administrator or contact RedHat support for more information