Kafka components are down and unable to process data, effectively bringing cluster to a down status.
Environment
Release: Component: AXAENG
Resolution
The issue seems like related to heap settings on the axa components (specially Transformer and Aggregator). Low heap setting likely causing issues with data processing. We made the adjustments in heap size and restarted axa components. It looks like AXA components are starting to processing data. Initially, there is still some Kafka lag , but after axa components continue run for few hours, Kafka lag has gone down and cluster performance is back to normal.