NSX Intelligence traffic flows are not visible in the UI

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Running NSX-T 3.2.1 onwards.
• The Policy API was used to configure TNs (Transport Nodes), that have since been force deleted.
• NSX Intelligence data collection does not get enabled on any newly added TNs.
• The existing TNs will continue to send NSX Intelligence data.
• On the Transport node, when you run the following nsxcli command, you see that the service is disabled:

get intelligence flows stats
Thu Jun 26 2023 UTC 09:15:50.916
             NSX Intelligence Host Flows Statistics
-----------------------------------------------------------------
Service is disabled.

• In the NSX-T manager log /var/log/proton/nsxapi.log we see the following entries:

2023-01-25T10:00:12.599Z ERROR workerTaskExecutor-43 GroupServiceImpl 4351 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM500012" level="ERROR" s2comp="grouping" subcomp="manager"] The path=[/infra/sites/default/enforcement-points/default/host-transport-nodes/<stale_node_display_name>] is invalid

• New hosts are not being added to the NSX Intelligence group:

Failed to process IntelligenceClusterConfig /infra/sites/default/intelligence/cluster-configs/2007f72b-0715-47da-ab93-f517d82cbc41:domain-c227759 : 

Environment

VMware NSX-T Data Center 4.x
VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

• When using the Policy API to configure transport nodes, the path to the transport node will contain the node's display name instead of its host UUID. 
• An internal NS group called the NSX Intelligence Data Collection group is created as a part of intelligence installation. This group at all times will store the policy paths of TNs that have data collection enabled on them. When a TN is force deleted, it is removed from the DB first, even if it is referenced elsewhere. Due to this force deletion, the policy path of the deleted TN remains in the group, leaving the group inconsistent.
• Adding new members to the Intelligence Data Collection group fails due to a failure in path validation for existing group members i.e. the force deleted host. Hence, data collection will not be enabled on any newly added TNs.
• When using the Manager API to configure transport nodes, the path to the transport node contains the node's UUID and this problem is not encountered.

Resolution

This is a known issue impacting VMware NSX.

Workaround:
Use the following steps to clean up the Data Collection group so that it contains the valid policy path, this will disable NSX intelligence data collection for the cluster, thus removing the system group used and then recreate the data collection and a new system group containing all the hosts again.

1. Fetch the NSX Application Platform (NAPP) registration results using the following API request.

GET /policy/api/v1/infra/sites/napp/registration

2. Set the the NSX Intelligence enablement to false using the following API request and the cluster_id info from the registration API request results (step 1). Make sure to have "is_intelligence_enabled": false" in the PATCH API request payload.

PATCH /policy/api/v1/infra/sites/napp/registration/<cluster_id> { "cluster_id" : "<cluster_id>", "is_intelligence_enabled" : false }

3. Set the NSX Intelligence enablement back to true using the same API. Make sure to have "is_intelligence_enabled": true" in the PATCH API request payload.

PATCH /policy/api/v1/infra/sites/napp/registration/<cluster_id> { "cluster_id" : "<cluster_id>", "is_intelligence_enabled" : true }