Data Services Manager unable to send logs to a configured syslog server.

Products

• VMware Data Services Manager

Issue/Introduction

Data Services Manager may be unable to establish a connection to a configured syslog server resulting in being unable to send logs to it.

This issue may occur inconsistently e.g. it is possible to have multiple environments on the same DSM version with only some impacted.

Environment

Data Services Manager 2.2.X

Cause

Inconsistent behaviour in fluent-bit service may result in incorrect certificate information being used to establish connection resulting in connection failure to the syslog server.

fluent-bit.log may log the following TLS-related error message:
container_name/fluent-bit[XXXX]: [YYYY/MM/DD HH:MM:SS] [error] [tls] error: unexpected EOF
container_name/fluent-bit[XXXX]: [YYYY/MM/DD HH:MM:SS] [error] [output:syslog:syslog.0] no upstream connections available

Resolution

This issue is resolved in DSM 9.0.0 and later releases.

Workaround for this issue is to apply the following:

First, manually connect to the syslog server in question to confirm the stored certificate information and TLS version used are correct (e.g. to rule out an unrelated configuration issue resulting in the connection failure):
# openssl s_client -connect <syslog-server-FQDN>:<port> -showcerts -CAfile /etc/pki/tls/certs/ca-bundle.crt

Edit this configuration file using vi or another text editor:
# vi /opt/vmware/tdm-provider/fluentbit-service/fluent-bit.conf

Replacing this line:
tls.ca_file             /host/etc/ssl/certs
With the following:
tls.ca_file             /fluent-bit/etc/ca-bundle.crt

Edit this YAML file using vi or another text editor:
# vi /opt/vmware/tdm-provider/docker-images/system-docker-compose.yml
After line 'volumes:', add the following line (including leading '- '):
- /etc/pki/tls/certs/ca-bundle.crt:/fluent-bit/etc/ca-bundle.crt:ro

Edit this configuration file using vi or another text editor:
# vi /opt/vmware/tdm-provider/fluentbit-service/fluent-bit-syslog.conf

Replacing this line:
tls.ca_file             /host/etc/ssl/certs
With the following:
tls.ca_file             /fluent-bit/etc/ca-bundle.crt

Restart fluent-bit service:
# systemctl restart fluent-bit.service

Remove and re-add the configuration of the impacted syslog server via the DSM UI.

Note: if DSM is upgraded to a later version prior to one including the fix (e.g. to versions 2.2.2 or 2.2.3) then the above workaround will have to be re-applied after upgrade.