Fluent Bit unable to forward logs from TKGm workload cluster to Elasticsearch due to ConfigMap reconciliation
Article ID: 422036
Updated On:
Products
Issue/Introduction
In a production Tanzu Kubernetes Grid (TKGm) environment, Fluent Bit failed to forward logs from the workload cluster (TKC) to the external Elasticsearch server.
Although all Fluent Bit pods were in the Running state, the logs were not visible in the Grafana dashboard that relies on Elasticsearch as the data source.
Environment
-
Product: Tanzu Kubernetes Grid (TKGm)
-
Version: 2.5.2
-
Component: Fluent Bit (deployed via Carvel package)
-
Integration: Elasticsearch and Grafana
-
Cluster Type: Production Workload Cluster (TKC)
Cause
The issue occurred because the Fluent Bit ConfigMap lacked several parameters required for Elasticsearch compatibility, such as:
-
Suppress_Type_Name -
Type -
Trace_Error -
Trace_Output
Even after adding these parameters manually, the configuration changes were reverted. This happened because the Kapp Controller automatically reconciled the ConfigMap to its original state based on the package’s predefined specification.
Resolution
To make the configuration changes persistent:
-
Update the
values.yamlfile of the Fluent Bit package with the required Elasticsearch parameters (Suppress_Type_Name,Type,Trace_Error,Trace_Output). -
Reapply the updated package using Tanzu CLI or through the Carvel kapp-controller workflow.
This ensures that the updated configuration is stored within the managed package definition and will not be overwritten during reconciliation.
Additional Information
-
Fluent Bit in TKGm 2.5.2 is a managed Carvel package. Directly editing its ConfigMap or DaemonSet does not persist changes because the kapp-controller enforces the package’s original configuration.
-
To apply any custom configuration, modifications must always be made via the package values file followed by a redeploy.
-
Reference Documentation:
Feedback
