Fluent Bit - How to Enable Log Forwarding to Syslog Server in vSphere Kubernetes Service (VKS) Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Users may want to configure Fluent Bit to send container logs to a remote Syslog server for centralized log collection and monitoring.

Environment

vSphere Kubernetes Service

Cause

By default, the Fluent Bit package does not forward logs to an external Syslog endpoint unless explicitly configured. Users must modify the fluent-bit-data-values.yaml file to enable and define the desired Syslog output configuration.

Resolution

To forward logs to a Syslog server using Fluent Bit in a TKG Service Cluster on vSphere Supervisor, follow the steps below:

Prepare the fluent-bit-data-values.yaml
Create or modify the fluent-bit-data-values.yaml file with the following Syslog output configuration:

---
fluent_bit:
  config:
    outputs: |
      [OUTPUT]
        Name   syslog
        Match  *
        Host   <HOST_IP>
        Port   <HOST_PORT>
        Mode   <TCP_MODE>
        Syslog_Message_key   message

Install or Update Fluent Bit Package
Use the following command to install Fluent Bit with your custom configuration:

tanzu package install fluent-bit \
  --package-name fluent-bit.tanzu.vmware.com \
  --version <version> \
  --values-file fluent-bit-data-values.yaml

Or to update an existing Fluent Bit package:

tanzu package installed update fluent-bit \
  --package-name fluent-bit.tanzu.vmware.com \
  --version <version> \
  --values-file fluent-bit-data-values.yaml

 

Verify the Deployment
Ensure the Fluent Bit pods are running and that logs are being received by the external Syslog server.

Verify this by running kubectl get pods -n <fluentbit-namespace>

Additional Information

Reference:
Refer to the official Fluent Bit package documentation for additional output plugins or advanced filtering:
Fluent Bit Package Reference - Broadcom TechDocs

To identify the source cluster of the logs, a record modifier can be added accordingly:

Reference document: 

Advanced Configuration for Fluent-bit Add-On

Logging Windows Worker Workloads

[FILTER]
        Name                record_modifier
        Match               *
        Record              tca_cluster_name wc0

  [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_Under        kubernetes

Use the default filter of type Kubernetes in fluent_bit.config.filters value, add a filter of type nest and a filter of type record_modifier to process the native logs so that the logs can be easily filtered out and displayed pretty on syslog server.

Remember to replace the test_cluster_name wc0 to your cluster name in record_modifier filter.