UMA not able to scrape Prometheus metrics "Failed to open the connection to the Prometheus endpoint".

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

We have deployed the new UMA (25.1.1.26) one of our OpenShift clusters but we don't appear to be pulling in the Prometheus metrics.
 
The following annotations have been added:
prometheus.io/scrape: "true"
prometheus.io/port: 9103
 
Able to see the metrics inside the pod if we do a curl http://localhost:9103/metrics, but can not see them in APM.
 
Checked a few of the app-container-monitor pods and they have the following error:
 
Failed to open the connection to the Prometheus endpoint :http://##.##.###:8080/metrics

We are not sure why it is trying to connect to the endpoint via port 8080 when the annotation has port 9103 specified. 

 

from UMA Logs:

3/11/25 08:30:33 AM GMT [INFO] [IntroscopeAgent.ConnectionThread] Connected to example.com:443,com.wily.isengard.client.transport.websocket.SSLWebSocketFactory in allowed mode.
Failed to open the connection to the Prometheus endpoint :http://##.##.###:8080/metrics

Environment

UMA (25.1.1.26)

CP: 25.1.1

Resolution

 The agent manager credential was missing on a few pods. Once updated the deployment and it appears to be working now.