Log Collector pod uses excessive memory and threads on Kubernetes

Products

• DX SaaS
• DX OI SaaS
• DX Operational Intelligence
• DX Operational Observability

Issue/Introduction

A Log Collector pod deployed from the container image consumes far more memory than its actual log volume would justify, and under a memory based Horizontal Pod Autoscaler it scales out even when incoming traffic is very low. Thread count for the pod is observed in the range of 90 to 300 or more, while the pod is processing only single digit events per second.

Inspecting the pod shows the thread count tracks the number of processor cores on the underlying Kubernetes node, not the processor allocation configured for the pod. Restarting the pod, rolling the deployment, or increasing the memory limit does not change the behavior. The extra memory is consumed by native thread stacks and related overhead, so it appears immediately at pod startup and is independent of log volume.

Environment

DX Operational Observability, SaaS.

Resolution

Set the LS_JAVA_OPTS environment variable on the Log Collector container so that the Java virtual machine reports the processor count you intend the pod to use. This corrects the count at the source, so every automatically sized thread pool inside the pipeline is affected at once. No change to the image and no change to logcollector.conf is required.

Path to change

Edit the Kubernetes manifest that deploys the Log Collector, at the following location in the object.

spec.template.spec.containers[] .env

Apply it to the container entry that runs the caapm/logcollector image. In a sidecar deployment this is the Log Collector container inside the application workload, not the application container.

The stanza to add, with the standard YAML indentation preserved.

env:

• name: LS_JAVA_OPTS
  value: "-XX:ActiveProcessorCount=4"

If the deployment is managed by a Helm chart, add the same variable wherever the chart exposes environment variables for the Log Collector container in the values file, rather than editing the rendered manifest.

To apply it directly to a running deployment for a quick test, the following command sets the same variable and triggers a rolling restart. Note that a value applied this way is lost the next time the manifest is reapplied from source control, so make the manifest or values file change as well.

kubectl set env deployment/<logcollector-deployment> -c <logcollector-container> LS_JAVA_OPTS=-XX:ActiveProcessorCount=4

Choosing the value

Set the number to the processor allocation you intend the pod to have, matching the processor request or limit declared for that container. A value of 4 was used in the validated case. Do not set it to 1, as this leaves no headroom for the pipeline. If the pod already declares an enforced processor limit and the thread count still tracks the node, the quota is not being enforced and this setting is the correct remedy.

Verifying the change

Confirm the variable reached the container:

kubectl exec <pod> -c <logcollector-container> -- env | grep LS_JAVA_OPTS

Confirm the thread count dropped, comparing against the value recorded before the change:

kubectl exec <pod> -c <logcollector-container> -- sh -c 'ls /proc/1/task | wc -l'

Confirm the pipeline is still healthy by checking the Log Collector pod log for startup errors, and confirm that log events are still arriving in the tenant. Then observe resident memory for the pod over a normal traffic period and confirm that the Horizontal Pod Autoscaler is no longer scaling out on memory pressure.