Logs stopped ingesting to tenant

Products

• DX OI SaaS

Issue/Introduction

Teams reporting that since yesterday logs many namespaces are not getting ingested to tenant.

Log quota is not exhausted. Not seeing any issues at the fluentd level.

We are seeing very limited logs from couple of containers like kibana, dashboards, etc. But majority of the logs from other namespaces, containers are missing.

Resolution

In another defect, Dev identified that in a rare scenario sometimes the ingestion limit reset didn't execute causing the logs to drop. But other than that all these tenants are frequently reaching their ingestion limits and being throttled. Only during midnight of UTC time the limits get reset and log ingestion starts.