NSX Application Platform Kafka topic input rate shows 0

Products

• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Kafka topic input rate keeps showing 0 in the NAPP monitor log for more than 15 minutes.
In 420 Dashboard UI, System -> NSX Application Platform -> Metrics -> KAFKA AVERAGE MESSAGE INPUT RATE shows 0

You can see the below error:
.v.n.k.m.s.MonitorServiceImpl: INTELLIGENCE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Kafka Input rate of topic active_flow is 0

when using the following process:
 

1. ssh to the NSX manager node
2.  Get monitor pod name by the command:
   napp-k get pod -l app.kubernetes.io/component=monitor
   • Sample output:
     ~# napp-k get pod -l app.kubernetes.io/component=monitor
     NAME                       READY   STATUS    RESTARTS   AGE
     monitor-7c4469b8b7-hc24d   1/1     Running   0          27h
3. Check monitor logs by the command:
   napp-k logs <monitor-pod-name> -f |  grep "Kafka Input rate of topic active_flow is"
   Note: This command will keep logging the input rate every minutes.
   • Sample output:
     ~# napp-k logs monitor-7c4469b8b7-hc24d -f | grep "Kafka Input rate of topic active_flow is"
     2024-07-23T22:09:35,965 INFO  [ThreadPoolTaskScheduler-2] c.v.n.k.m.s.MonitorServiceImpl: INTELLIGENCE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Kafka Input rate of topic active_flow is 0
     2024-07-23T22:10:38,018 INFO  [ThreadPoolTaskScheduler-7] c.v.n.k.m.s.MonitorServiceImpl: INTELLIGENCE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Kafka Input rate of topic active_flow is 0
     2024-07-23T22:11:35,965 INFO  [ThreadPoolTaskScheduler-2] c.v.n.k.m.s.MonitorServiceImpl: INTELLIGENCE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Kafka Input rate of topic active_flow is 0
     2024-07-23T22:12:35,965 INFO  [ThreadPoolTaskScheduler-6] c.v.n.k.m.s.MonitorServiceImpl: INTELLIGENCE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Kafka Input rate of topic active_flow is 0
     2024-07-23T22:13:38,027 INFO  [ThreadPoolTaskScheduler-10] c.v.n.k.m.s.MonitorServiceImpl: INTELLIGENCE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Kafka Input rate of topic active_flow is 0
     2024-07-23T22:14:35,966 INFO  [ThreadPoolTaskScheduler-8] c.v.n.k.m.s.MonitorServiceImpl: INTELLIGENCE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Kafka Input rate of topic active_flow is 0
4. Wait for 10~15 minutes. Check if all input rates shows 0 in the log in the last 10~15 minutes. To stop the log, please press "control + c" on the keyboard

Environment

NAPP 4.2.0
Will be fixed in a future release

Cause

The monitor cronjob to update Kafka metrics crashed at some point. As a result, while Kafka is working properly, monitor pod cannot log the latest input rate and pending messages from Kafka. On 420 UI, KAFKA AVERAGE MESSAGE INPUT RATE will show 0.
After restarting the monitor pod, the new monitor pod will start the cronjob, and all Kafka logs in monitor pod will be back to normal.

Resolution

From the NSX Manager CLI, while logged in as root, Perform the following command to restart the monitor pod.

• napp-k rollout restart deployment monitor