"Error: Metric Store error" is shown in App Metrics UI

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Error: Metric Store error was shown in App Metrics UI within few minutes of accessing the dashboard as well as some red graphs. 



Troubleshooting steps

1. From the metric-store-cf-auth-proxy.stderr.log, we can see error message 'failed CAPI request with 401(Unauthorized)'.

{"level":"info","timestamp":"2022-10-18T03:00:19.083Z","app":"cf-auth-proxy","message":"CAPI request returned non-200","url":"/v3/apps","status_code":"401"}
{"level":"error","timestamp":"2022-10-18T03:00:19.084Z","app":"cf-auth-proxy","message":"failed to make request","error":"failed CAPI request (/v3/apps) with status: 401 (Unauthorized)"}
{"level":"error","timestamp":"2022-10-18T03:29:04.159Z","app":"cf-auth-proxy","message":"failed to read from Oauth2 server","error":"token is expired, exp = 2022-10-18 03:20:32 +0000 UTC"}

2. From metric-store.stderr.log, we can see a lot of dropped points for remote node messages also.
3. The problem is not with Admin user but some specific user
4. Metric Store version is v1.5.2

Root cause

Metrics Store retrieves data via CAPI V3 queries which requires doppler.firehose or logs.admin permissions in UAA scope. Admin user usually has required permission, but specific user should have doppler.firehose but actually doesn't in this case. 

Environment

Product Version: 2.11

Resolution

The issue exists in Metric Store v1.5.2 and earlier releases. It's fixed in v1.5.3 as described in the release note.

• Metric Store v1.5.3 fixed issue where the charts were not loading when the doppler.firehose was not in scope

Please upgrade to Metric Store v1.5.3+ to resolve the issue.