No new metrics in Developer Portal after upgrading from 5.0 to 5.1

Products

• CA API Developer Portal

Issue/Introduction

After upgrading the Developer Portal from 5.0 to 5.1, the MySQL database has been externalized and is no longer using the internal Postgres database.

All metrics (druid) data have been successfully migrated but since the start of the new version, the metrics from new requests are no longer shown in the GUI.

The druid container logs does not show a obvious error.

Environment

API Portal 5.1

Resolution

1. Run the following commands in the coordinator container:

curl http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics/status 

curl http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics_hour/status 

2. Verify if the status is running and there are no reset errors like:

{"timestamp":"2023-06-15T21:12:12.111Z","exceptionClass":"org.apache.kafka.common.KafkaException","message":"Failed to construct kafka consumer","streamException":false}

If there are errors, run the following command to reset the tasks in the coordinator container:

curl -X POST 'http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics/reset'

curl -X POST 'http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics_hour/reset'

NOTE: The POST /druid/indexer/v1/supervisor/<supervisorId>/reset operation clears stored offsets, causing the supervisor to start reading offsets from either the earliest or latest offsets in Kafka (depending on the value of useEarliestOffset). After clearing stored offsets, the supervisor kills and recreates any active tasks, so that tasks begin reading from valid offsets.

Resetting the supervisor may cause Kafka messages to be skipped or read twice, resulting in missing or duplicate data.