Grafana Dashboard Shows Negative Success Rate After Pod Restarts

Products

• Symantec Identity Security Platform - IDSP (formerly VIP Authentication Hub)

Issue/Introduction

You observe negative values for Success Rates (%) under the Integration sections (e.g., SPI for SMS, Email, IVR, or Risk) in your monitoring dashboard. Additionally, specific metrics like SMS may appear in red despite having a 100% success rate.

Environment

Symantec Identity Security Platform (IDSP) 4.x
IDSP-v10-Monitoring-Dashboard Grafana dashboard

Cause

This issue occurs because the application pods (such as AZserver) restarted during an upgrade or maintenance event. When pods restart, the Prometheus/metric counters reset to zero. If the selected Grafana time range spans across this reset event, the calculation for the success rate fails to handle the counter reset properly, resulting in a negative value.

Resolution

You must reduce the Grafana time range to exclude the period when the pod restarts occurred.

To resolve the display issue, follow these steps:

•     Open your Grafana Dashboard.
•     Locate the Time Range picker in the top-right corner of the interface.
•     Adjust the start time to a point after the pods stabilized and the upgrade completed.
•     Refresh the dashboard to confirm the Success Rate metrics display accurate, non-negative values.

By ensuring the query range does not include the counter reset point, the dashboard performs calculations based on a continuous set of data.