EDR: Troubleshooting Rsyslog Configuration

Products

Issue/Introduction

Confirm rsyslog settings and check connectivity.

Environment

• EDR Server:  All supported
• Rsyslog

Resolution

Steps to troubleshoot rsyslog from the EDR server to a remote rsyslog server.

1.  Confirm rsyslog port is open.

netstat -anp | grep LISTEN | grep <port>  (confirm rsyslog is listening on the port)

To test an open a port on the rsyslog server.
ncat -ul 1234 (where 1234 is the syslog udp port)

2.  Confirm firewalld or iptables allow the traffic on EDR and the syslog server.

On EDR:
     /usr/share/cb/cbcheck firewall -a
     iptables -nL | grep <port>

On rsyslog server
     iptables -nL | grep <port>

3.  Test connectivity.   Send data from EDR to the rsyslog server on the designated port.

On the rsyslog server:
     tcpdump -Xnni <interface> <port> proto 17

Send data from the EDR server:
      /usr/share/cb/cbsyslog -l  (lists the different types) 
      /usr/share/cb/cbsyslog -vfe feed.storage.hit.process
      /usr/share/cb/cbsyslog -vfe watchlist.hit.process
      or send a test file
      ncat -vu --send-only <rsyslog IP> <port> < <filename>

4.  Look for errors in rsyslogd.

rsyslogd -dn 2> /dev/null  | less

5.  Check if the logs are appearing on the EDR server.

tail -f /var/log/cb/notifications/cb-notifications-test.log

6a.  If the events are not appearing on the remote rsyslog server, add spooling to cb-coreservices.conf

if $programname startswith 'cb-notifications-' then -?
DynaFile;CbSyslogStandardFormatWithPID
$WorkDirectory /var/lib/rsyslog  # location of spoolfiles on the disk
$ActionQueueFileName cbtest   # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
& @192.168.10.252:1234;CbSyslogStandardFormatWithPID 
& ~

6b. Restart rsyslog.

systemctl restart rsyslog

7.  For testing purposes, consider adding the remote syslog server to /etc/rsyslog.conf.  Add the following line in the 'forwarding rule' section.  Restart rsyslog service.

*.* @<rsyslog IP>:<port>

 

Additional Information

• If rsyslog is using TCP, remember to use @@ instead of @.
• To configure EDR to rsyslog events, refer to the EDR Integration Guide.  Quick notes below.

cd /etc/rsyslog.d/
cp cb-coreservices.conf cb-coreservices.conf-orig
vi cb-coreservices.conf
(Per the guide, add the following to the end of the first $programname == 'cb-notifications' line)
& @<remote IP address>:<UDP port>;CbLogFormatWithPID & ~

• If possible, it is more efficient to use Event Forwarding syslog feature.  Why?  Event Forwarding listens on the RabbitMQ bus.