Alertmanager and prometheus-operator pods don't start and keep showing ImagePullBackOff and Init:ErrImagePull upgrading to binamilegacy in IDSP

Products

• Symantec Identity Security Platform - IDSP (formerly VIP Authentication Hub)

Issue/Introduction

Upgrading Prometheus and Grafana deployment to use bitnamilegacy repository, the alertmanager and prometheus-operator pods don't start and keep showing errors:

  ImagePullBackOff
  Init:ErrImagePull

The following commands have been used:

# helm upgrade prometheus-operator bitnami/kube-prometheus -n monitoring -f prometheus-values.yaml --version=11.3.10 
# helm upgrade grafana-operator bitnami/grafana-operator -n monitoring -f grafana-values.yaml  --version=4.9.37  

Resolution

Stop both pods before doing the upgrade to fix the issue:

// 1. Scale to 0

# kubectl scale statefulset prometheus-prometheus-operator-kube-p-prometheus --replicas=0 -n monitoring
# kubectl scale statefulset alertmanager-prometheus-operator-kube-p-alertmanager --replicas=0 -n monitoring

// 2. Wait for pods to terminate

# kubectl wait --for=delete pod alertmanager-prometheus-operator-kube-p-alertmanager-0 prometheus-prometheus-operator-kube-p-prometheus-0 -n monitoring --timeout=300s

// 3. Delete the StatefulSet (PVCs remain)

# kubectl delete statefulset prometheus-prometheus-operator-kube-p-prometheus -n monitoring
# kubectl delete statefulset alertmanager-prometheus-operator-kube-p-alertmanager -n monitoring

## 4. Helm upgrade recreates it

# helm upgrade prometheus-operator bitnami/kube-prometheus -f prometheus-values.yaml -n monitoring
# helm upgrade grafana-operator bitnami/grafana-operator -f grafana-values.yaml -n monitoring