Syslog Server not Receiving Logs from Elastic Runtime

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

This article provides a troubleshooting guide for the cases when you are unable to receive logs from Elastic Runtime to the Syslog.

Syslog Server does not receive any logs from Elastic Runtime.

 

Environment

Cause

Following are the possible reasons for why you are not receiving any logs from Elastic Runtime:

• syslog_forwarder logs may not be configured properly
• Network failure with Syslog server
• Syslog server might be receiving the logs but not displaying them as expected

 

Resolution

Following are the troubleshooting steps you can take to check your configuration and connection with your Syslog Server:

1. bosh ssh or ssh into any of the job VMs i.e. UAA or Cloud Controller
2. Run the command nc -vv -u <syslog remote host> <syslog remote port> to check the connection to the Syslog server. A successful connection would result in Connection to <syslog remote host><syslog remote port> port [udp/syslog] succeeded! If it did not succeed, please check with your network administrator to see if there are any factors that may be hindering the connection
3. If the nc command is successful, check conf using /etc/rsyslog.d/00-syslog_forwarder.conf and see if the right Syslog server is configured on the line *.* @@<Syslog-hostname-here>;CfLogTemplate

   $WorkDirectory /var/vcap/sys/rsyslog/buffered # where messages should be buffered on disk
   
   # Forward vcap messages to the aggregator
   #
   $ActionResumeRetryCount -1              # Try until the server becomes available
   $ActionQueueType LinkedList             # Allocate on-demand
   $ActionQueueFileName agg_backlog        # Spill to disk if queue is full
   $ActionQueueMaxDiskSpace 32m            # Max size for disk queue
   $ActionQueueLowWaterMark 2000           # Num messages. Assuming avg size of 512B, this is 1MiB.
   $ActionQueueHighWaterMark 8000          # Num messages. Assuming avg size of 512B, this is 4MiB. (If this is reached, messages will spill to disk until the low watermark is reached).
   $ActionQueueTimeoutEnqueue 0            # Discard messages if the queue + disk is full
   $ActionQueueSaveOnShutdown on           # Save in-memory data to disk if rsyslog shuts down
   
   ....
   
   *.* @@<Syslog-hostname-here>;CfLogTemplate
   

4. If the configuration does not reflect the correct Syslog server, then edit the file with the correct server details.
5. Finally, check if there are packets sent to Syslog server by sudo tcpdump -n "dst host <syslog-server-host> and dst port<syslog remote port>". If there are packets sent to the Syslog server, check with your Syslog server's support/documents to see why it's not being displayed. If no packets are being sent, please contact Pivotal Support.