Running Prometheus queries in Healthwatch using the API
search cancel

Running Prometheus queries in Healthwatch using the API

book

Article ID: 297941

calendar_today

Updated On:

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

When using Healthwatch, it can be useful to manually run PromQL queries against the prometheus process running on the TSDB VM. This article will explain how to do this, and give a few examples of queries and their results.

Environment

Product Version: 2.1

Resolution

The first step is to ssh to a TSDB VM in the Healthwatch deployment. Once there, the command for running the query would be: 
curl -k https://localhost:9090/api/v1/query?query=<query> --cacert
/var/vcap/jobs/prometheus/config/certs/prometheus_ca.pem --cert
/var/vcap/jobs/prometheus/config/certs/prometheus_certificate.pem --key
/var/vcap/jobs/prometheus/config/certs/prometheus_certificate.key
  • Port 9090 is the default port that prometheus runs on.
  • /api/v1/query is the API endpoint used to run queries against, as described here: https://prometheus.io/docs/prometheus/latest/querying/api/#expression-queries
You then need to add a PromQL query into the <query> field, in order to return the data. For example: 
curl -k https://localhost:9090/api/v1/query?query=system_mem_percent --cacert
/var/vcap/jobs/prometheus/config/certs/prometheus_ca.pem --cert
/var/vcap/jobs/prometheus/config/certs/prometheus_certificate.pem --key
/var/vcap/jobs/prometheus/config/certs/prometheus_certificate.key
This command returns the stored metric for the Percentage of memory used on each VM in each deployment. Taking one example: 
{"metric":{"__name__":"system_mem_percent","deployment":"cf-f9f5fc19cfeaa1c05bd1","exported_job":"diego_cell","id":"a26be528-b6f5-40d6-b4bf-3f184be93ac1","index":"a26be528-b6f5-40d6-b4bf-3f184be93ac1","instance":"<ip>:9090","instance_group":"diego-cell","ip":"<ip>","job":"healthwatch-pas-exporter","origin":"system_metrics_agent","product":"VMware Tanzu Application
Service","scrape_instance_group":"pas-exporter-gauge","source_id":"system_metrics_agent","system_domain":"<domain>","unit":"Percent"},"value":[1690368072.166,"15.43155520802843"]}
  • "deployment", "exported_job" and "id" show the BOSH Deployment name and instance type/name. In this case, it's a diego cell in the cf deployment.
  • "ip" is the IP of the diego cell VM.
  • value is displayed in two fields:
    • 1690368072.166 (The time in epoch)
    • 15.43155520802843  (The percentage of memory used on the VM)

Another example would be if you wish to run a more complicated PromQL query: 
curl -k https://localhost:9090/api/v1/query --data-urlencode
'query=label_replace((sum by (placement_tag)(CapacityTotalMemory) - sum by
(placement_tag)(CapacityRemainingMemory)) / sum by (placement_tag
(CapacityTotalMemory), "placement_tag", "cf", "placement_tag", "") ' --cacert
/var/vcap/jobs/prometheus/config/certs/prometheus_ca.pem --cert
/var/vcap/jobs/prometheus/config/certs/prometheus_certificate.pem --key
/var/vcap/jobs/prometheus/config/certs/prometheus_certificate.key
This format allows you to collect data using more PromQL operators.

The main list of KPI used by TAS are found here:

https://docs.vmware.com/en/VMware-Tanzu-Application-Service/2.13/tas-for-vms/monitoring-kpi.html

Powered by Wolken Software