ZABPROC STC shutting down by itself

Products

• SYSVIEW Performance Management

Issue/Introduction

How to fix issue with ZABPROC ending with RC000 and below messages.

18.29.01 STC98548  BPXM023I (ZABUSER) ZABM005E z/OS APM Bridge is stopping due to connection issues with APM Kafka 
18.29.01 STC98548  BPXM023I (ZABUSER) ZABM006I z/OS APM Bridge has stopped 
...
 2024-05-20 21:28:01.690 WARN  c.b.mfd.zab.sink.KafkaSinkService        - Attempt 4: Exception while sending heartbeat message. Send
 failed; nested exception is org.apache.kafka.common.errors.TimeoutException: Topic zabhealth not present in metadata after 60000 ms.
 2024-05-20 21:29:01.692 WARN  c.b.mfd.zab.sink.KafkaSinkService        - Attempt 5: Exception while sending heartbeat message. Send
 failed; nested exception is org.apache.kafka.common.errors.TimeoutException: Topic zabhealth not present in metadata after 60000 ms.
 2024-05-20 21:29:01.693 INFO  com.broadcom.mfd.zab.ZABMonitor          - Received event: SINK_DOWN and ZAB State: com.broadcom.mfd.z
 ab.ZABMonitor$DisconnectZDMState@f05aca19
 2024-05-20 21:29:01.695 INFO  com.broadcom.mfd.zab.ZABMonitor          - ZAB is in <<ShutdownZABState>>
 2024-05-20 21:29:01.695 INFO  c.b.mfd.zab.console.ConsoleMessages      - Available Locales
 2024-05-20 21:29:01.702 INFO  c.b.mfd.zab.console.ConsoleMessages      - Available Locales
 2024-05-20 21:29:01.712 INFO  o.a.kafka.common.metrics.Metrics         - Metrics scheduler closed
 2024-05-20 21:29:01.713 INFO  o.a.kafka.common.metrics.Metrics         - Closing reporter org.apache.kafka.common.metrics.JmxReporte
 r
 2024-05-20 21:29:01.713 INFO  o.a.kafka.common.metrics.Metrics         - Metrics reporters closed
 2024-05-20 21:29:01.714 INFO  o.a.kafka.common.utils.AppInfoParser     - App info kafka.producer for producer-1 unregistered
 2024-05-20 21:29:01.714 WARN  o.s.b.f.s.DisposableBeanAdapter          - Invocation of destroy method failed on bean with name 'prod
 ucerFactory': org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException

Cause

Prereq Kafka topics are not exist 

Resolution

Based on error message from description above ZABPROC is trying to make a write to zabhealth Kafka topic but it not exist, after 1 hour ZABPROC start shutdown process. 

ZAB requires 2 Kafka topics to work properly, 

1. zab
2. zabhealth

In case they not exist ZABPROC will attempt to create them but this ability depends on Kafka configuration, topic creation by agents like ZAB could be forbidden. In order to fix it Kafka configuration need to be checked, in this particular case it was on DX APM application side.