We are not able to see any analytics data on API Developer Portal 5.1.2 on Kubernetes after deployment
We deployed API Developer Portal on Kubernetes using terraform.
Release : 5.1.2
The deployment was created from our helm charts in terraform with some customization .
We do not see analytics data all pods seems to be running but the the Minio pod keeps restarting .
The minio pod logs says it expect 4 nodes and can not connect to all of them which is causing the healthcheck probe to restart the pod while it waits for all nodes to come online.
Unable to connect to http://xxxxxx-caapim-portal.xxx.xxx:9000/opt/data: Post http://minio-x.minio.sharedapps-caapimxxx.xxxx:9000/minio/storage/v8/opt/data/getinstanceid?: dial tcp: lookup xxxxxxx-caapim-xxx.xxx on 172.20.0.10:53: no such host
The client deployed on Kubernetes using Terraform .
The client did set the replica count for Minio to 3 while the default is 1 .
The helm chart is deploying a single node Minio when the replica count is set to 1 , in all other cases it will create a multi node minio which has a default of 4 nodes regardless of the replica count set
so the only valid settings for replica count for Minio is 1 or 4.
After increasing the replica count to 4 the Minio pods started correctly and stay started
Due to problems with the minio pods for a while , we needed to reset the supervisors in the coordinator pod to restart the analytic metric collection .
2023-07-21T14:08:57,527 ERROR [KafkaSupervisor-apim_metrics] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Failed to initialize after 20 retries, aborting. Please resubmit the supervisor spec to restart this supervisor [KafkaSupervisor-apim_metrics]: {class=org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor
Connect to the coordinator pod .and run the following two commands :
curl -X POST http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics/reset
curl -X POST http://localhost:8081/druid/indexer/v1/supervisor/apim_metrics_hour/reset