vDefend SSP Alarm: Remote syslog server logging is down

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

• You are running SSP 5.0 and later.
• You are seeing an alarm with the description "Remote syslog server logging is down."

Environment

vDefend SSP >= 5.0

Cause

• Network Connectivity Issues: The Fluentd pod cannot reach the remote syslog server due to network disruptions or misconfigurations.
• Remote Server Downtime: The remote syslog server itself could be experiencing downtime.

Resolution

Check Network Connectivity:

• Ensure firewalls, routing rules, and DNS configurations allow traffic from the SSPI VM to the remote server.

Review Fluentd Logs:

• Check Fluentd pod logs (kubectl logs <fluentd-pod> -n nsxi-platform) from SSPI VM as root for errors related to the syslog server connection.
• Possible error in logs in fluentd pod, 

  "#0 failed to emit data to remote syslog",
"#0 failed to write data into buffer by buffer overflow action",
"#0 failed to flush the buffer",
"#0 buffer is full",

Check Remote Server Health:

• Confirm the remote syslog server is up, reachable, and accepting connections.
• Make sure the rsyslog config and ports are open and not blocked by the firewall. 
• For a TLS-based connection, validate using OpenSSL from the SSPI VM as root:

  openssl s_client -connect <remote_syslog_server>:<tls-port> -CAfile <syslog server ca_certificate.pem file> -showcerts -msg
  
  This pem file will be uploaded in SSP UI while configuring the syslog server. This will be stored in the Fluentd pod at 
  
  k -n nsxi-platform exec -it fluentd-0 -- bash
cat /opt/bitnami/fluentd/syslog/ca-cert.pem
   

• For TCP/UDP connection: 
  Steps to see if it's connected via TCP/UDP: 
  Log in to the cluster-api pod from the SSPI VM:
  
  k exec -it deploy/cluster-api -c cluster-api -- bash
  
  and execute the command.
  
  nc -zvu <remote_syslog_server_ip> 514

Additional Suggested Action: Please try removing the existing Syslog configuration and re-adding it to verify if log transmission resumes successfully.

Before performing this step, ensure that the support bundle is collected and all above mentioned diagnostic actions are completed.

This will enable you to provide comprehensive data to Broadcom Support for potential root cause analysis.