vDefend SSP Alarm: Remote syslog server logging is down

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

• You are running SSP 5.0 and later.
• You are seeing an alarm with the description "Remote syslog server logging is down."

Environment

vDefend SSP >= 5.0

Cause

• Network Connectivity Issues: The Fluentd pod cannot reach the remote syslog server due to network disruptions or misconfigurations.
• Remote Server Downtime: The remote syslog server itself could be experiencing downtime.

Resolution

    
Check Network Connectivity:

• Ensure firewalls, routing rules, and DNS configurations allow traffic from SSPI VM to the remote server.

Review Fluentd Logs:

• Check Fluentd pod logs (kubectl logs <fluentd-pod> -n nsxi-platform) from SSPI VM as root for errors related to the syslog server connection.
• Possible error in logs in fluentd pod, 

  "#0 failed to emit data to remote syslog",
"#0 failed to write data into buffer by buffer overflow action",
"#0 failed to flush the buffer",
"#0 buffer is full",

Check Remote Server Health:

• Confirm the remote syslog server is up, reachable, and accepting connections.
• Make sure the rsyslog config and ports are open and not blocked on firewall. 
• For TLS based connection, validate using openssl from SSPI VM as root:

  openssl s_client -connect <remote_syslog_server>:<tls-port> -CAfile <syslog server ca_certificate.pem file> -showcerts -msg
  
  This pem file will be uploaded in SSP UI while configuring syslog server. This will be stored in fluentd pod at 
  
  k -n nsxi-platform exec -it fluentd-0 -- bash
cat /opt/bitnami/fluentd/syslog/ca-cert.pem
   

• For TCP/UDP connection: 
  Steps to see if its connected via TCP/UDP: 
  Login into cluster-api pod from SSPI VM:
  
  k exec -it deploy/cluster-api -c cluster-api -- bash
  
  and execute the command.
  
  nc -zvu <remote_syslog_server_ip> 514