Not able to see any analytics on API Developer Portal 5.1.2 on Kubernetes

Products

• CA API Developer Portal

Issue/Introduction

We are not able to see any analytics data on API Developer Portal 5.1.2 on Kubernetes after deployment 

We deployed API Developer Portal on Kubernetes using terraform.

Environment

Release : 5.1.2

Cause

The deployment was created from our helm charts in terraform with some customization  .

We do not see analytics data all pods seems to be running but the the Minio pod keeps restarting .

The minio pod logs says it expect 4 nodes and can not connect to all of them which is causing the healthcheck probe to restart the pod while it waits for all nodes to come online.

Unable to connect to http://xxxxxx-caapim-portal.xxx.xxx:9000/opt/data: Post http://minio-x.minio.sharedapps-caapimxxx.xxxx:9000/minio/storage/v8/opt/data/getinstanceid?: dial tcp: lookup xxxxxxx-caapim-xxx.xxx on 172.20.0.10:53: no such host

Resolution

The client deployed on Kubernetes using Terraform .

The client did set the replica count  for Minio to 3 while the default is 1 .

The helm chart is deploying a single node Minio when the replica count is set to 1 , in all other cases it will create a multi node minio which has a default of 4 nodes regardless of the replica count set

so the only valid settings for replica count for Minio is 1 or 4.

After increasing the replica count to 4 the Minio pods started correctly and stay started 

Due to problems with the minio pods for a while , we needed to reset the supervisors in the coordinator pod to restart the analytic metric collection .

2023-07-21T14:08:57,527 ERROR [KafkaSupervisor-apim_metrics] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Failed to initialize after 20 retries, aborting. Please resubmit the supervisor spec to restart this supervisor [KafkaSupervisor-apim_metrics]: {class=org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor

Connect to the coordinator pod .and run the following two commands :

curl -X POST  http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics/reset
curl -X POST  http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics_hour/reset