Flow integration allows you to consume and analyze flow records produced by a third-party device (for example, a switch).
For information about configuring the flow integration refer to the Flow collection tab topic in the NSX Lastline Defender Portal User Guide.
Symptoms:
We recommend using a dedicated sensor for processing third-party flows. If you have done so, you can simply check that there are flows from that specific sensor. There are a couple of ways of doing it:
source
property value.AAAAAAAAAAAAAAAAAAAA:flow-collector
, you will need to select the netflow-*
index pattern, and then run the following query: source:"AAAAAAAAAAAAAAAAAAAA:flow-collector"
(note that you must enclose the license in double quotes or a syntax error will result because of the unescaped colon ":" character).If you know that the ingested records include some specific values (for example, you expect flows from host A to host B), try searching for those records. You can do this in a couple of ways:
netflow.src_ip:A AND netflow.dst_ip
src_ip:A AND dst_ip:B
You have enabled the integration and want to confirm that records are processed correctly: here are some recommended steps to do this.
The relevant log files on the sensor ingesting the netflows are:
llcollector logs: /var/log/llcollector/netflow-v5_2055_udp/llcollector.worker.silk_netflow_worker.process-?.log
(the name will vary depending on the specific collector configuration)
When netflow processing is working as expected, you should see messages similar to the following:
2019-08-14 08:04:52,474 - llcollector.worker.silk_netflow_worker.process-1/process-1[188973]/MainThread - INFO - Starting to process netflow file '/var/run/llcollector/netflow-v5_2055_udp/silk_netflow_collector/data/20190814080336_collector.4ipyhq'. 2019-08-14 08:04:52,653 - llcollector.worker.silk_netflow_worker.process-1/process-1[188973]/MainThread - INFO - Silk netflow processor report: I0814 08:04:52.532678 20257 main.cpp:989] Silk netflow processor, version 1.0.28 I0814 08:04:52.532784 20257 main.cpp:621] Processing input file /var/run/llcollector/netflow-v5_2055_udp/silk_netflow_collector/tmp/20190814080336_collector.4ipyhq I0814 08:04:52.599521 20257 main.cpp:715] Successfully parsed 14349 netflow records (new: 7631, known: 6718) in 0 seconds while ignoring 0 invalid records I0814 08:04:52.646335 20257 main.cpp:805] Successfully submitted SensorUpload fa1ae7d8-fa28-4baf-a926-be0c175546eb (789424 bytes) embedding 7631 netflow records to llshed I0814 08:04:52.646981 20257 main.cpp:938] Finished handling 1 input file(s) in 0 seconds, processing 14349 valid netflow records in total (14349 netflow records per second, 46% aggregation rate) I0814 08:04:52.646996 20257 main.cpp:945] Statistics: processing_time=0 parsing_time=0 uploading_time=0 files_success=1 files_failure=0 uploads_success=1 uploads_failure=0 netflows_success=14349 netflows_failure=0 netflows_new=7631 netflows_known=6718 aggregation_percent=46 netflows_per_second=14349 2019-08-14 08:04:52,659 - llcollector.worker.silk_netflow_worker.process-1/process-1[188973]/MainThread - INFO - Finished processing netflow file '/var/run/llcollector/netflow-v5_2055_udp/silk_netflow_collector/data/20190814080336_collector.4ipyhq'
llshed upload: /var/log/llshed/llshed.worker.llcollector_papi_sensorupload_netflows_worker_?.log
2019-08-14 08:04:43,151 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-5 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI. 2019-08-14 08:05:08,403 - llshed.aggregators/llcollector_papi_sensorupload_netflows_worker_1[174962]/MainThread - INFO - Requesting information for 217 locations at lltic. 2019-08-14 08:05:08,449 - llshed.aggregators/llcollector_papi_sensorupload_netflows_worker_1[174962]/MainThread - INFO - Finished requesting information for 217 locations at lltic. 2019-08-14 08:05:21,441 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-4 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI. 2019-08-14 08:05:35,372 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-1 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI. 2019-08-14 08:06:08,401 - llshed.aggregators/llcollector_papi_sensorupload_netflows_worker_1[174962]/MainThread - INFO - Requesting information for 196 locations at lltic. 2019-08-14 08:06:08,438 - llshed.aggregators/llcollector_papi_sensorupload_netflows_worker_1[174962]/MainThread - INFO - Finished requesting information for 196 locations at lltic. 2019-08-14 08:06:30,602 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-5 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI. 2019-08-14 08:06:42,996 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-4 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI. 2019-08-14 08:06:44,272 - llshed.processors/llcollector_papi_sensorupload_netflows_worker_1[174962]/Thread-2 - INFO - Successfully uploaded SensorUpload submission 'fa1ae7d8-fa28-4baf-a926-be0c175546eb' to PAPI.
If you do not see similar activity in these log files (even though netflows are being sent to the appliance) or observe unexpected errors, the netflow processing is not working correctly. Please contact VMware Technical Support .
You may notice some additional directories that are related to the flow processing. We list them here for completeness, but we suggest you ignore them during the troubleshooting and focus on log files instead:
silk_netflow_collector
: ignore it as it contains only error logs from a third-party tool called flowcap - the logs in that subdirectory are supposed to be empty.
/var/run/llcollector/netflow-v5_2055_udp/silk_netflow_collector/data/
contains the actual flow data