Campaigns may fail to be generated by NDR Campaign Correlation Pipeline

Products

VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

The NDR campaign correlation pipeline can sometimes fail to process events into campaigns. If the NDR UI shows a large number of events but no associated campaigns, this may indicate that it is experiencing this issue.

Environment

NAPP 4.2 and before.

Cause

This issue is caused when nsx-ndr-worker-detection-event-update-windower pod enters a failed state and is unable to recover on its own. To verify this is the issue, check the logs of the nsx-ndr-worker-detection-event-update-windower deployment by running the following command on the manager.

Access the NSX Manager (any one of the managers) in root mode.

napp-k get pods | grep ndr

malware-prevention-feature-switch-watcher-notifier-ndr-ccbhw9xk 1/1 Running 0 46h

nsx-ndr-api-6c4b6d4644-jv4c4 2/2 Running 0 46h

nsx-ndr-frontend-service-6b8c4d8d9c-75gsn 1/1 Running 0 46h

nsx-ndr-nsx-ndr-hooks-enable-ids-l2c5g 0/1 Completed 0 46h

nsx-ndr-service-minio-bucket-configuration-pcap-gt4j2 0/1 Completed 0 46h

nsx-ndr-service-partitioned-db-retention-28692360-ppbz8 0/1 Completed 0 24h

nsx-ndr-service-time-columns-db-retention-28692000-q5jbl 0/1 Completed 0 30h

nsx-ndr-service-workers-kafka-provisioning-s6zwl 0/1 Completed 0 46h

nsx-ndr-service-workers-s3-provisioning-lq9k2 0/1 Completed 0 46h

nsx-ndr-worker-campaign-manager-667dff9dc5-wq86l 1/1 Running 0 46h

nsx-ndr-worker-correlation-rule-runner-594ccb89fd-pxdwh 1/1 Running 0 46h

nsx-ndr-worker-correlation-task-matcher-7d5c995f9b-hj66q 1/1 Running 0 46h

nsx-ndr-worker-detection-event-aggregator-64fffdc994-jpnvw 1/1 Running 0 46h

nsx-ndr-worker-detection-event-scorer-5dd795c85c-l9glb 1/1 Running 1 (38h ago) 46h

nsx-ndr-worker-detection-event-update-windower-86b5d8bfdd-m5j8w 1/1 Running 0 46h

nsx-ndr-worker-enriched-ids-event-translator-844cb6cbd6-62phh 1/1 Running 0 46h

nsx-ndr-worker-file-event-translator-b84bf954c-7n6ff 1/1 Running 0 46h

nsx-ndr-worker-nta-event-translator-7f98bc7fdd-57h82 1/1 Running 0 46h

nsx-ndr-worker-pcap-storer-5cc48f778d-jnr8f 1/1 Running 0 46h

nsx-ndr-worker-siem-notification-scheduler-7b468cf6d4-cj8zc 1/1 Running 0 46h

nsx-ndr-worker-siem-notification-sender-84976ccb8b-wp8dm 1/1 Running 1 (38h ago) 46h

3. napp-k logs pod/nsx-ndr-worker-detection-event-update-windower-XXXXXXXXXX-XXXXX | grep 'Aiokafka has not sent fetch request for'

[2024-07-20 16:20:41,214] [1] [ERROR] [^---AIOKafkaConsumerThread]: Aiokafka has not sent fetch request for TP(topic='ndr-detection-update', partition=0) since start (started 7.77 hours ago)

...

[2024-07-22 15:44:41,405] [1] [ERROR] [^---AIOKafkaConsumerThread]: Aiokafka has not sent fetch request for TP(topic='ndr-detection-update', partition=0) since start (started 2.30 days ago)

If those logs has above errors, then that indicates this issue is present. Alternatively, the logs may be found in the support bundle, if collected.

Resolution

To resolve the issue,

Access the NSX Manager (any one of the managers) in root mode.

2. Restart the nsx-ndr-worker-detection-event-update-windower pod deployment using the following command;

napp-k rollout restart deploy nsx-ndr-worker-detection-event-update-windower

3. Monitor the NDR UI to confirm successful campaign generation after new events are observed. This may take up to 10 minutes after the events are seen in the UI. Please note that not all events will generate campaigns.