Event log forwarded by Fluent bit in VKS guest cluster is invisible in Aria Operations for Logs

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Fluent bit has been installed in VKS guest cluster with the steps in Install Fluent Bit Package.
• Fluent bit pods are in running status without exception logs output with the command:
  
  kubelet logs -n tanzu-system-logging <fluent-bit-pod-name>
  
  
• The Fluent bit data values are from A Sample of fluent-bit-data-values.yaml for Log Forwarding to VCF Operations for Logs with Output configuration of syslog:
  
      outputs: |
      [OUTPUT]
        Name   syslog
        Match  *
        Host   <aria-operations-for-logs-ip>
        Port   514
        Mode   <udp or tcp>
        Syslog_format  rfc5424
        Syslog_Message_key   message
  
  
• The events from the Control plane node of VKS guest cluster are only visible when specify the source with the CP node address but the events are truncated with only time stamp. For example:
  
  May 12, 2026   2026-05-12T10:10:32.406840Z - - - - -
18:10:32.665   source  event_type  facility  priority  hostname  appname  procid  msgid

Environment

VMware Aria Operations for Logs
VMware vSphere Kubernetes Service

Cause

The empty messages are caused by a strict key mismatch in the Fluent Bit syslog output configuration. It is configured to exclusively use the lowercase message field as the log payload (Syslog_Message_key message). However, the input sources do not contain this exact key. Systemd logs use an uppercase MESSAGE field. API Audit logs are complex JSON structures that lack a message field entirely.

Resolution

1. Change the Fluent bit data values:
   
   
   • Change parser from json to key:message for apiserver_audit:
     
     [INPUT]
  Name              tail
  Tag               apiserver_audit.*
  Key               message 
  Path              /var/log/kubernetes/kube-apiserver.log
  DB                /var/log/flb_kube_audit.db
  Mem_Buf_Limit     50MB
  Refresh_Interval  10
  Skip_Long_Lines   On
     
     
   • Add new filter for systemd log:
     
     [FILTER]
  Name                modify
  Match               kube_systemd.*
  Rename              MESSAGE message
2. Update the Fluent bit. For example:
   
   vcf package installed update fluent-bit --values-file fluent-bit-data-values-modified.yaml -n fluent-bit