Prometheus pod consumes high CPU usage

Products

• VIP Authentication Hub

Issue/Introduction

We are observing high CPU usage of prometheus-prometheus-operator-prometheus-0 pod in monitoring namespace

" kubectl top pods -n monitoring " shows the following result

NAME                                                      CPU(cores)   MEMORY(bytes)
alertmanager-prometheus-operator-alertmanager-0           1m           29Mi
prometheus-operator-grafana-785c67994d-tm9xw              1m           235Mi
prometheus-operator-kube-state-metrics-7c75dff95c-9lq5q   2m           25Mi
prometheus-operator-operator-84c69665f5-p9t5v             1m           45Mi
prometheus-operator-prometheus-node-exporter-926vh        5m           19Mi
prometheus-operator-prometheus-node-exporter-lf9c2        2m           15Mi
prometheus-operator-prometheus-node-exporter-pkdvq        1m           17Mi
prometheus-prometheus-operator-prometheus-0               211m         745Mi

Environment

Release : VIP AuthHub ssp 1.0.3211 (2022.Oct.06)

Resolution

Edit the prometheus instance object (run kubectl edit prometheus -n monitoring) and add the resources section under the spec, as shown below

resources:       limits:         cpu: 1000m       requests:         cpu: 400m