Some NTA Detectors Fail to Detect Events when Deploying NDR without Intelligence

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Certain NTA detectors (Vertical Port Scan, Horizontal Port Scan, LLMNR / NBNTS, Uncommonly Used Port, Traffic Drop, and Remote Services) require flow data to be persisted for later analysis. If other detectors are also failing, or other symptoms are being observed besides missing NTA events, the root cause may be different and these steps provided may not completely address the issue.

 

Validate this by logging into the NSX Manager and using napp-k :

napp-k exec -it $(napp-k get pods | awk '/cluster-api-/ { print $1 }') -c cluster-api -- bash

In the cluster api pod, run:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka:9092 \
  --command-config /root/adminclient.props \
  --topic correlated_flow_viz \
  --describe
 
Example output:

Error while executing topic command : Topic 'correlated_flow_viz' does not exist as expected
[2024-07-30 19:04:06,946] ERROR java.lang.IllegalArgumentException: Topic 'correlated_flow_viz' does not exist as expected
    at org.apache.kafka.tools.TopicCommand.ensureTopicExists(TopicCommand.java:215)
    at org.apache.kafka.tools.TopicCommand.access$700(TopicCommand.java:78)
    at org.apache.kafka.tools.TopicCommand$TopicService.describeTopic(TopicCommand.java:559)
    at org.apache.kafka.tools.TopicCommand.execute(TopicCommand.java:108)
    at org.apache.kafka.tools.TopicCommand.mainNoExit(TopicCommand.java:87)
    at org.apache.kafka.tools.TopicCommand.main(TopicCommand.java:82)
 (org.apache.kafka.tools.TopicCommand)

If the command returns an error saying that the topic does not exist, you have confirmed that the topic is missing.

Environment

NAPP 4.2.0

 

Cause

A certain kafka topic (correlated_flow_viz) that is required for persisting flow data is only created when Intelligence is installed, so it is missing in the configuration when NDR is installed without Intelligence.

Resolution

(1) Log into the NSX Manager and using napp-k , login to cluster-api pod

napp-k exec -it $(napp-k get pods | awk '/cluster-api-/ { print $1 }') -c cluster-api -- bash

(2) In the cluster api pod, get the number of partitions for the new topic:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka:9092 \
  --command-config /root/adminclient.props \
  --topic correlated_flow \
  --describe |
  awk 'NR==1 {print $5, $6}'
 
Example output:
PartitionCount: 3

(3) In the cluster api pod, create the topic using the number of partitions from the previous step:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka:9092 \
  --command-config /root/adminclient.props \
  --topic correlated_flow_viz \
  --create \
  --partitions <partition count should match the number from step 3> \
  --replication-factor 1 \
  --config retention.ms=3600000 \
  --config retention.bytes=10737418240
 
Example output:N/A (if no stack trace or errors are output, the operation is successful)

(4) In the nsx manager, restart the latestflow deployment:

napp-k rollout restart deployment/latestflow
After this, the correlated_flow_viz table in druid should be present after flows have made their way through the processing pipeline.