DXO2 Service view showing all services in unknown state.

Products

• DX Operational Observability

Issue/Introduction

All services in DXO2 have been found in unknown state, grayed out. Checking kafka pod following errors have been captured:

ERROR [kafka-producer-network-thread | producer-7]: [c.c.d.s.t.i.k.KafkaPublisher] Error while ingesting message to kafka, Sent:{"tenantId":"###","task":"###","frequency":1,"timestamp":1782812156800,"totalService":0}:Expiring 3 record(s) for ###_tenant_service_process-5:120000 ms has passed since batch creation on topicName:###_tenant_service_process
org.apache.kafka.common.errors.TimeoutException: Expiring 3 record(s) for ###_tenant_service_process-5:120000 ms has passed since batch creation
ERROR [kafka-producer-network-thread | producer-7]: [c.c.d.s.t.i.k.KafkaPublisher] Error while ingesting message to kafka, Sent:{"tenantId":"###","task":"###","frequency":1,"timestamp":1782812156800,"totalService":0}:Expiring 3 record(s) for ###_tenant_service_process-5:120000 ms has passed since batch creation on topicName:###_tenant_service_process

Environment

26.1 on-prem.

Cause

Kafka data structure damaged or corrupted.

Resolution

Contact Broadcom Support. Kafka data needs to be rebuilt.