No new metrics in Developer Portal after upgrading from 5.0 to 5.1

Products

• CA API Developer Portal

Issue/Introduction

We upgraded our Developer Portal from 5.0 to 5.1 . In addition, we have moved the database to an external mysql database and therefore no longer use the internal postgres database.
All metrics (druid) data has been successfully migrated. But since the start of the new version, metrics of new requests are no longer shown in the GUI.

The druid container logs do not show a obvious error.

Environment

Release : 5.1

Resolution

run the following commands in the coordinator container

curl http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics/status 

curl http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics_hour/status 

Verify if the status is running , and there are no reset errors like 

{"timestamp":"2023-06-15T21:12:12.111Z","exceptionClass":"org.apache.kafka.common.KafkaException","message":"Failed to construct kafka consumer","streamException":false}

If so run the following command to reset the tasks in the coordinator container 

curl -X POST 'http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics/reset'

curl -X POST 'http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics_hour/reset'

Note this leads to - The POST /druid/indexer/v1/supervisor/<supervisorId>/reset  operation clears stored offsets, causing the supervisor to start reading offsets from either the earliest or latest offsets in Kafka (depending on the value of useEarliestOffset). After clearing stored offsets, the supervisor kills and recreates any active tasks, so that tasks begin reading from valid offsets.
Resetting the supervisor may cause Kafka messages to be skipped or read twice, resulting in missing or duplicate data