NetOps Portal Event Manager Publication Service shows Failed in System Status

Products

• Network Observability
• CA Performance Management

Issue/Introduction

The DX NetOps Portal System Status page shows the Event Manager Publication Service is in a Failed state.

Why is the Event Manager Publication Service failing?

The Event Manager Publication Service shows "Lost contact with event publication ...".

Configured Portal to send events to the NetOps Kafka cluster following these steps in the Install NetOps Kafka documentation.

Seeing errors like this for the kafka cluster hosts in the $PORTAL_HOME/CA/PerformanceCenter/EM/log/EMService.log file.

INFO  | kafka-admin-client-thread | adminclient-1 | 2024-01-30 18:38:02,497 | org.apache.kafka.clients.NetworkClient                           
      | [AdminClient clientId=adminclient-1] Node -2 disconnected.
WARN  | kafka-admin-client-thread | adminclient-1 | 2024-01-30 18:38:02,497 | org.apache.kafka.clients.NetworkClient                           
      | [AdminClient clientId=adminclient-1] Connection to node -2 (<node2_HostName>/<node2_IP>:9092) could not be established. Broker may not be available.
INFO  | kafka-admin-client-thread | adminclient-1 | 2024-01-30 18:38:03,299 | org.apache.kafka.clients.NetworkClient                           
      | [AdminClient clientId=adminclient-1] Node -3 disconnected.
WARN  | kafka-admin-client-thread | adminclient-1 | 2024-01-30 18:38:03,299 | org.apache.kafka.clients.NetworkClient                           
      | [AdminClient clientId=adminclient-1] Connection to node -3 (<node2_HostName>/<node1_IP>:9092) could not be established. Broker may not be available.
INFO  | kafka-admin-client-thread | adminclient-1 | 2024-01-30 18:38:03,300 | org.apache.kafka.clients.NetworkClient                           
      | [AdminClient clientId=adminclient-1] Node -1 disconnected.
WARN  | kafka-admin-client-thread | adminclient-1 | 2024-01-30 18:38:03,300 | org.apache.kafka.clients.NetworkClient                           
      | [AdminClient clientId=adminclient-1] Connection to node -1 (<node2_HostName>/<node3_IP>:9092) could not be established. Broker may not be available.

Environment

All supported DX NetOps Portal releases integrated with a netops-kafka cluster.

Cause

The kafka cluster hosts have selinux configured in PERMISSIVE mode.

Resolution

Set selinux to be fully disabled on the kafka cluster hosts and reboot the hosts. Some instances show the need to also restart the EM (caperfcenter_eventmanager) service.