UMA does not collect prometheus metrics from certain pod.

Products

• DX SaaS
• DX Operational Observability

Issue/Introduction

UMA agent cannot scrape the metrics following annotations from certain pod endpoint ending with log sequence shown below and found as part of cluster-performance-prometheus-ng pod:

time="###-###-###" level=warning msg="scraping prometheus metrics: http://###.###.###.###:###/###/### (example-pod)" component=Scraper error="Get \"http://###.###.###.###:###/###/###": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

 

 

Cause

Environmental - network policy prevented connection.

Resolution

Ensure that your annotated pod is exposing prometeus data and that network policy is not preventing the connection. 

For instance, when pod has been annotated as below:

prometheus.io/scrape: true

prometheus.io/port: 9600

prometheus.io/path: /test/path

Log in to the affecred pod and execute the following:

curl localhost:9600/test/path and see if there is data. In case there is data in line with Prometeus format and data is not scraped check network policies.