How to Monitor and Alert on DX SaaS Log Ingestion Limits to Prevent Clamping

Products

• DX SaaS

Issue/Introduction

• Log ingestion suddenly stops, and no new logs appear in the dashboard.
• Data remains "clamped" (dropped) until midnight UTC.
• Users are unaware they have reached their daily ingestion quota (e.g., 2GB or 5GB limits) until data loss has already occurred.
• There is a need to meter or throttle ingestion at the O2 layer to avoid hitting hard limits.

Environment

• Product: DX Operational Intelligence (DX OI) / DX SaaS (O2)
• Component: Log Collector / Log Analytics

Cause

DX SaaS uses daily log ingestion quotas. When a tenant reaches its Hard Limit, the system automatically clamps (stops) ingestion for the remainder of the day. There is currently no built-in feature to set "per-hour" metering or to throttle specific collectors at the O2 Tenant Administration level once data has reached the limit.

Resolution

While you cannot currently throttle ingestion at the O2 level, you can proactively manage volume and avoid clamping using the following methods:

1. Configure Proactive Alarms and Notifications

The system generates Out-of-the-Box (OOTB) alarms when ingestion thresholds are reached. You must create a Notification Policy to ensure these reach the Tenant Admin via email:

• Soft Limit Alarm: Triggered when the tenant is approaching its daily quota.
• Hard Limit Alarm: Triggered when the limit is hit and clamping begins.
• Action: Create a notification policy based on above alarms.

2. Monitor Usage by reviewing Metrics/Traces by Agents and Dashboards:

To see real-time consumption and identify high-volume:

• Navigate to Health Monitoring > Connector Health dashboard.
• Review metrics under Metrics/Traces by Agent to identify which specific collectors or log monitors are generating the most volume.

3. Control Volume at the Source

The most effective way to "meter" logs is at the ingestion source (Log Forwarder/Filebeat):

• Filtering: Refine your log monitor configurations to include only necessary log levels (e.g., ERROR/WARN instead of INFO/DEBUG).
• Exclusion Rules: Use regular expressions at the agent level to drop high-noise messages before they are transmitted to the DX SaaS environment.

4. Request a Quota Increase

If the current daily limit is insufficient for your business requirements (e.g., moving from a POC to a Production-scale), contact Broadcom Support to request an increase in the daily ingestion limit.