Fluent Bit does not forward some pod logs due to "[ warn] [http_client] cannot increase buffer” in Tanzu Kubernetes Grid

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

You will observe below symptoms :

• All fluent-bit pods are in RUNNING status.
• Other logs are forwarded to the destination correctly.
• But few logs are not processed due to a warning "[http_client] cannot increase buffer" in the fluent-bit pod logs. 

kubectl -n tanzu-system-logging logs fluent-bit-#####
#> [2024/07/30 02:07:01] [ warn] [http_client] cannot increase buffer: current=32000 requested=64768 max=32000

 

NOTE - Fluent Bit package has been deployed as a Tanzu Package. 

Environment

Tanzu Kubernetes Grid 1.x and 2.x 

Cause

When receiving a huge response from the Kubernetes API server, Fluent Bit cannot process it and does not forward it to the destination.

The issue was discussed upstream; please refer to https://github.com/fluent/fluent-bit/issues/3918.

Resolution

Re-configure the buffer size for HTTP client as a higher value than the requested size from the warning message observed in the Fluent Bit Pod logs. 
Below is a sample configuration; you can decide and modify Buffer_Size based on your requirements. 

[FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On
    Keep_Log Off
    K8S-Logging.Parser On
    K8S-Logging.Exclude On
    Buffer_Size 256KB # <---------- Target

For more details on configuration parameters, you can refer to the Fluent Bit Official document.

 

Additional Information

• Update a Running Fluent Bit Deployment
• Fluent Bit Unit Sizes
• Set the plugin's logging verbosity level as “debug” for detailed logs
• For Tanzu Kubernetes Grid Integrated Edition, you can refer to KB 298688