Prometheus server not sending alerts to Alert Manager in Tanzu Kubernetes Grid 1.2 and 1.3

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

• Prometheus and Alert Manager are deployed via the extension framework in Tanzu Kubernetes Grid (TKG) 1.2 or 1.3.
• Alerts are generated at the Prometheus server which can be observed from the "Alerts" section when Prometheus is exposed as LB or using an Ingress controller like Contour.
• Alert Manager does not receive the alerts and any slack or Email notification alert configured in Alert Manager does not trigger.

Environment

VMware Tanzu Kubernetes Grid 1.x
VMware Tanzu Kubernetes Grid Plus 1.x

Cause

 

Resolution

This is a known issue affecting TKG 1.2 and 1.3. There is currently no resolution.

Workaround:

You can work around this issue by modifying the Prometheus extension:

New installation of the Prometheus extension

1. For TKG 1.2 Extension in this step of "Prepare the Prometheus Extension Configuration File"

https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.2/vmware-tanzu-kubernetes-grid-12/GUID-extensions-prometheus.html#prepare-the-prometheus-extension-configuration-file-2

and similarly for TKG 1.3 for the same step

https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.3/vmware-tanzu-kubernetes-grid-13/GUID-extensions-prometheus.html#prepare-the-prometheus-extension-configuration-file-2

update the prometheus-data-values.yaml to include monitoring.prometheus_server.config.prometheus_yaml

This section can be copied from the file ~/tkg-extensions-v1.2.0+vmware.1/monitoring/prometheus/values.yaml or ~/tkg-extensions-v1.3.0+vmware.1/monitoring/prometheus/values.yaml for 1.2 and 1.3 version of extensions respectively.

 

2. After adding the contents of monitoring.prometheus_server.config.prometheus_yaml you  need to change the port number under monitoring.prometheus_server.config.prometheus_yaml.alerting.alertmanagers under targets and set the port number to 80 instead of 9093. The following is a sample configuration:

        alerting:
          alertmanagers:
          - scheme: http
            static_configs:
            - targets:
              - "prometheus-alertmanager.tanzu-system-monitoring.svc:80"

For reference, modified prometheus-data-values.yaml file is attached to this article.
 

3. Once this file is created you can proceed with creating the secret and deploying the Prometheus extension as outlined in the TKG 1.2 documentation (Deploy Prometheus on a Tanzu Kubernetes Cluster) or in the TKG 1.3 documentation (Deploy Prometheus on a Tanzu Kubernetes Cluster).

Update an existing Prometheus extension installation

Follow steps one and two in the previous section and then replace the existing secret using the following command (Note: the command assumes you are running it from the directory where the prometheus-data-values.yaml file exists):
 

kubectl create secret generic prometheus-data-values --from-file=values.yaml=monitoring/prometheus/vsphere/prometheus-data-values.yaml -n tanzu-system-monitoring -o yaml --dry-run=client | kubectl replace -f -

This will update the prometheus-server configmap. You may need to wait up to five minutes for kapp to reconcile based on the syncPeriod configured for the extension

Additional Information

You can verify if the port is changed by running the following command:

kubectl -n tanzu-system-monitoring get cm prometheus-server -o yaml | grep prometheus-alertmanager.tanzu-system-monitoring.svc
          - "prometheus-alertmanager.tanzu-system-monitoring.svc:80"
      targets:\n      - \"prometheus-alertmanager.tanzu-system-monitoring.svc:80\"\n  -

Additionally, alerts should be sent to Alert Manager from Prometheus.