Prometheus Application metrics are not working on UMA apart from istio metrics

Products

• DX SaaS
• DX Operational Observability

Issue/Introduction

Below is the deployment and application annotations

oc get deploy auditlogger -oyaml |grep -i prometheus
       prometheus.io/path: /metrics
       prometheus.io/port: "8180"
       prometheus.io/scrape: "true"

oc get pods auditlogger-###### -oyaml|grep -i promet
   prometheus.io/path: /stats/prometheus
   prometheus.io/port: "15020"
   prometheus.io/scrape: "true"
   - name: ISTIO_PROMETHEUS_ANNOTATIONS

Istio integration prometheus 

Environment

DX O2

Resolution

There is an issue with configMap for istio matcher rules. In order to see the metrics follow the below steps.

1. Delete the below annotations if any present on the application deployment
   1. ca.broadcom.com/prometheus.scraper.metric_prefix
   2. ca.broadcom.com/prometheus.scraper.include_metrics
      
      
2. Add the below annotations on the application deployment
   1. ca.broadcom.com/prometheus.scraper.metric_prefix : '[{".*":"{type}|{k8s_pod_name}"}]'
   2. ca.broadcom.com/prometheus.scraper.include_metrics : '[".*"]'

3. Wait for a minute or so, for the new annotations to pick up. Once the new annotations are read, users will be able to visualise the metrics. Below is the sample metric path

    <k8s_cluster_name>|<k8s_namespace_name>|<k8s_owner_name>|<type>|<k8s_pod_name>

   

Additional Information

DX Agents - Istio Service Mesh