Telemetry data is missing and usage reports are very slow to generate

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Elastic Application Runtime telemetry data is missing from your dashboards. The usage reports in apps manager may also take a long time or fail to generate.

The telemetry-centralizer VM shows a timeout connecting to the usage service:

Collecting data from Usage Service at https://app-usage.system.example.com
2026/08/13 04:31:13 deprecated: golang.org/x/oauth2: Transport.CancelRequest no longer does anything; use contexts
Error: Failed collecting from Usage Service: Failed retrieving app usage data: error retrieving app usages data: error accessing usage service: error performing request Get "https://app-usage.system.example.com/system_report/app_usages": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

The app-usage-server app in the system org / system space is returning 499 status codes after about 30 seconds (in the below example, response_time: 29.880934)

2026-08-13T12:53:49.258-07:00 [RTR/11] [OUT] app-usage.system.example.com - [2026-08-13T19:53:19.377267481Z] "GET /system_report/app_usages HTTP/1.1" 499 0 22 "-" "Go-http-client/1.1" "###.###.###.###:20586" "###.###.###.###:61058" x_forwarded_for:"###.###.###.###" x_forwarded_proto:"https" vcap_request_id:"###########" response_time:29.880934 gorouter_time:0.000151 app_id:"##########" app_index:"1" instance_id:"#########" x_cf_routererror:"endpoint_failure (context canceled)" x_b3_traceid:"############" x_b3_spanid:"########" x_b3_parentspanid:"-" b3:"##########"

Environment

Elastic Application Runtime

Cause

This is caused by slow mysql queries on the app_usage_service database. If using the internal mysql server, the slow query log may show slow queries similar to the following (see the "Schema" and "Query_time"):

SET timestamp=1786652614;
SELECT MAX(`org_space_instance_counts`.`service_instance_count`) FROM `org_space_instance_counts` WHERE `org_space_instance_counts`.`occurred_at` BETWEEN '2026-07-01 00:00:00' AND '2026-07-31 23:59:59' AND `org_space_instance_counts`.`org_guid` = '#########' AND `org_space_instance_counts`.`space_guid` = '########';
# Time: 2026-08-13T20:25:06.309410Z
# User@Host: app_usage[app_usage] @  [###.###.###.###]  Id: 11798538
# Schema: app_usage_service  Last_errno: 0  Killed: 0
# Query_time: 71.500304  Lock_time: 0.000003  Rows_sent: 1  Rows_examined: 3049030  Rows_affected: 0  Bytes_sent: 109
# Tmp_tables: 0  Tmp_disk_tables: 0  Tmp_table_sizes: 0
# InnoDB_trx_id: 0
# Full_scan: No  Full_join: No  Tmp_table: No  Tmp_table_on_disk: No
# Filesort: No  Filesort_on_disk: No  Merge_passes: 0
#   InnoDB_IO_r_ops: 0  InnoDB_IO_r_bytes: 0  InnoDB_IO_r_wait: 0.000000
#   InnoDB_rec_lock_wait: 0.000000  InnoDB_queue_wait: 0.000000
#   InnoDB_pages_distinct: 8184

Resolution

The Usage Service queries and database are being optimized. The fix for this issue is expected to ship with the September 2026 release of Elastic Application Runtime