NSX Network Detection and Response - Number of messages in db.events.xxxx: xxxxxxx exceeds threshold

Products

• VMware vDefend Network Detection and Response

Issue/Introduction

Symptoms:

• Lastline_test_appliance reported the below ERROR

2023-04-11 01:02:40,087: output: > HARDWARE: OK
2023-04-11 01:02:40,087: output: > NETWORK: OK
2023-04-11 01:02:40,087: output: > SOFTWARE:
2023-04-11 01:02:40,087: output: >   FAILURE: Number of messages in db.events.xxxx: 2361131 exceeds threshold: 10000
2023-04-11 01:02:40,087: output: > Max total number of messages: 2362004 exceeds threshold: 100000
2023-04-11 01:02:40,087: output: Exiting with error-code 3
2023-04-11 01:02:40,177: output:
2023-04-11 01:02:40,178: Running lastline_test_appliance: return code: 3

• The Manager nodes RabbitMQ backlog was piling up when checked with the command "rabbitmqctl -p llq_v1 list_queues | grep -v -w 0$"

2023-04-11 01:11:11,973: Executing rabbitmqctl -p llq_v1 list_queues | grep -v -w 0$ as current user
2023-04-11 01:11:12,944: output: Listing queues
2023-04-11 01:11:12,944: output: db.events.xxxx 2367864
2023-04-11 01:11:12,944: output: db.analysis_completed.xxxx    33
2023-04-11 01:11:12,944: output: db.inserted_pcaps.xxxx 266
2023-04-11 01:11:12,944: output: hunter.e2a5752709e26fd201886e95db95deb0        1
2023-04-11 01:11:12,944: output:
2023-04-11 01:11:12,944: Return code: 0

• Restarting docker service did not help with resolving the issue
• db.events kept increasing

Resolution

We can try clear the queue and then try to increase the number of worker nodes however please contact to Broadcom support team to look into this.