Request URI Too Long in Grafana dashboards

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

You see "Request URI Too Long" message in one or more dashboards, and no data is shown.

This happens when the http method configured is GET and the prometheus query is too long for that method.

Resolution

This is issue is resolved in Healthwatch 2.2.x. This version introduces Grafana 8.x (see see HealthWatch release notes ) that changes the default HTTP method for Prometheus data source from GET to POST (see  Grafana release notes breaking changes).

If you can't upgrade at the moment, as a workaround you can change the grafana config file ( can't be done in the dashboard), following these steps.

1. bosh ssh into the grafana VM, in the healthwatch deployment.
2. run "sudo -i" to become root
3. locate /var/vcap/data/jobs/grafana/<combination-of-numbers-and-letters>/config/provisioning/datasources/prometheus.yml and make a backup
4. edit prometheus.yml in the above location, locate the line with "httpMethod: GET" and change it to "httpMethod: POST"
5. save the file and exit the editor
6. run "monit restart grafana" to restart the service. The "HTTP Method" setting in the dashboard should have been changed from GET to POST