Troubleshooting the flow integration

Products

• VMware

Issue/Introduction

Flow integration allows you to consume and analyze flow records produced by a third-party device (for example, a switch).

For information about configuring the flow integration refer to the Flow collection tab topic in the NSX Lastline Defender Portal User Guide.

Symptoms:

Check that records have been indexed

We recommend using a dedicated sensor for processing third-party flows. If you have done so, you can simply check that there are flows from that specific sensor. There are a couple of ways of doing it:

• On the Dashboard: Network page, select the sensor that is ingesting the flow records, and check the Network and security summary widget: it should report that some netflow records have been processed. 
• If you have access to the Network Explorer page, check that there are flows that have the sensor's license as their source property value.
  Assuming the sensor's license is AAAAAAAAAAAAAAAAAAAA:flow-collector, you will need to select the netflow-* index pattern, and then run the following query: source:"AAAAAAAAAAAAAAAAAAAA:flow-collector" (note that you must enclose the  license in double quotes or a syntax error will result because of the unescaped colon ":" character).
  For sensors dedicated to ingesting flow records, there should be some netflow records and no records for other record types (such as webrequest, pdns) since the sensor is monitoring no traffic.

Look for specific records

If you know that the ingested records include some specific values (for example, you expect flows from host A to host B), try searching for those records. You can do this in a couple of ways:

• On the Network analysis page , search for netflow.src_ip:A AND netflow.dst_ip
• If you have access to the Network explorer page , search the netflow index in Kibana with a query like src_ip:A AND dst_ip:B

Cause

Troubleshooting the integration

You have enabled the integration and want to confirm that records are processed correctly: here are some recommended steps to do this.

Resolution

Check the Sensor logs

The relevant log files on the sensor ingesting the netflows are:

• llcollector logs: /var/log/llcollector/netflow-v5_2055_udp/llcollector.worker.silk_netflow_worker.process-?.log (the name will vary depending on the specific collector configuration)

  When netflow processing is working as expected, you should see messages similar to the following:

  2019-08-14 08:04:52,474 - llcollector.worker.silk_netflow_worker.process-1/process-1[188973]/MainThread - INFO - Starting to process netflow file '/var/run/llcollector/netflow-v5_2055_udp/silk_netflow_collector/data/20190814080336_collector.4ipyhq'.
  2019-08-14 08:04:52,653 - llcollector.worker.silk_netflow_worker.process-1/process-1[188973]/MainThread - INFO - Silk netflow processor report:
  
  I0814 08:04:52.532678 20257 main.cpp:989] Silk netflow processor, version 1.0.28
  I0814 08:04:52.532784 20257 main.cpp:621] Processing input file /var/run/llcollector/netflow-v5_2055_udp/silk_netflow_collector/tmp/20190814080336_collector.4ipyhq
  I0814 08:04:52.599521 20257 main.cpp:715] Successfully parsed 14349 netflow records (new: 7631, known: 6718) in 0 seconds while ignoring 0 invalid records
  I0814 08:04:52.646335 20257 main.cpp:805] Successfully submitted SensorUpload fa1ae7d8-fa28-4baf-a926-be0c175546eb (789424 bytes) embedding 7631 netflow records to llshed
  I0814 08:04:52.646981 20257 main.cpp:938] Finished handling 1 input file(s) in 0 seconds, processing 14349 valid netflow records in total (14349 netflow records per second, 46% aggregation rate)
  I0814 08:04:52.646996 20257 main.cpp:945] Statistics: processing_time=0 parsing_time=0 uploading_time=0 files_success=1 files_failure=0 uploads_success=1 uploads_failure=0 netflows_success=14349 netflows_failure=0 netflows_new=7631 netflows_known=6718 aggregation_percent=46 netflows_per_second=14349
  
  2019-08-14 08:04:52,659 - llcollector.worker.silk_netflow_worker.process-1/process-1[188973]/MainThread - INFO - Finished processing netflow file '/var/run/llcollector/netflow-v5_2055_udp/silk_netflow_collector/data/20190814080336_collector.4ipyhq'
  

• llshed upload: /var/log/llshed/llshed.worker.llcollector_papi_sensorupload_netflows_worker_?.log

  When netflow processing is working as expected you should see messages similar to the following:

  2019-08-14 08:04:43,151 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-5 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI.
  2019-08-14 08:05:08,403 - llshed.aggregators/llcollector_papi_sensorupload_netflows_worker_1[174962]/MainThread - INFO - Requesting information for 217 locations at lltic.
  2019-08-14 08:05:08,449 - llshed.aggregators/llcollector_papi_sensorupload_netflows_worker_1[174962]/MainThread - INFO - Finished requesting information for 217 locations at lltic.
  2019-08-14 08:05:21,441 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-4 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI.
  2019-08-14 08:05:35,372 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-1 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI.
  2019-08-14 08:06:08,401 - llshed.aggregators/llcollector_papi_sensorupload_netflows_worker_1[174962]/MainThread - INFO - Requesting information for 196 locations at lltic.
  2019-08-14 08:06:08,438 - llshed.aggregators/llcollector_papi_sensorupload_netflows_worker_1[174962]/MainThread - INFO - Finished requesting information for 196 locations at lltic.
  2019-08-14 08:06:30,602 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-5 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI.
  2019-08-14 08:06:42,996 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-4 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI.
  2019-08-14 08:06:44,272 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-2 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI.
  

If you do not see similar activity in these log files (even though netflows are being sent to the appliance) or observe unexpected errors, the netflow processing is not working correctly. Please contact VMware Technical Support .

Other files

You may notice some additional directories that are related to the flow processing. We list them here for completeness, but we suggest you ignore them during the troubleshooting and focus on log files instead:

• silk_netflow_collector: ignore it as it contains only error logs from a third-party tool called flowcap - the logs in that subdirectory are supposed to be empty.

• /var/run/llcollector/netflow-v5_2055_udp/silk_netflow_collector/data/ contains the actual flow data

Additional Information

Impact/Risks:
If this integration is not working as expected, NTA data will not display in the UI or in the Network Explorer interfaces.