Accuracy issue with aggregation queries

Products

• DX SaaS

Issue/Introduction

The issue is observed on the dashboard.

We have started facing the accuracy issue with aggregation queries 
Considering one single metric, we get non-zero value for last 1 day but the value becomes empty when we move last 2 days.

Example -For crucial APIs, we define one metric name prefix each.

When an API is invoked by the user, we emit two metrics -

1.  indicating whether the API succeeded or not and
2.  what was the time taken in millis.

For each API invocation, the metric name is suffixed by the request ID and metric attributes specific to API inputs are also added.

So essentially, each API invocation results in registering two new metrics. Suffixing metric name with request ID is needed because otherwise the attributes are overwritten when the same API is later invoked with different inputs resulting in different metric attributes for the same metric name prefix.

   

Environment

DX SAAS  25.11.1

Resolution

Metadata generally does not guarantee the retrieval of all records that exist only within a specific range, which appears relevant to the majority of the records in this case. Two suggested options to help address this:

1.  Limit Range to 1 Day: Searching with a range of one day or less will search for all records and should return the expected results. Technically, this uses a 25-hour window to ensure a full day fits within the search.
   
   1-Day (or shorter) ranges: Searching within a range of one day or less queries all forks within that period.
   This should return all metrics that were live during that day.
   
   
2. Use "ForceRange" Query Hint: You can include a query hint in the metric query: queryHints: [{"hint" : "ForceRange"}]. This instructs the query planner to use an index for the range search. While this ensures all results are returned, evaluation may be slower, and the query will fail with an exception if the planner cannot utilize the index. 

- ForceRange Hint: This hint only works for searches where the endTime is set to "now" (or 0). It is not designed for historical searches. Additionally, when this hint is used, the standard "all forks" search for 1-day ranges is bypassed.

 

Recommendation is to try redefining the aggregation queries to base on actual metric data rather than metadata and restructuring the metric submissions to generate metric path to include tenantid and siteid info instead of using attributes to store these info. 

This approach will help mitigate the inconsistent or empty query results potentially from data gaps.