Flow collection stops for a vCenter data source and re-selecting NetFlow does not restore flows

Products

VCF Operations for Networks

Issue/Introduction

Flow collection stops for one or more vCenter data sources in VMware VCF Operations for Networks (formerly vRealize Network Insight). The data source continues to show as connected, and configuration collection stays current, but the Flows column on the Data Sources page remains empty and no flow-based data appears in the user interface.

Editing the existing data source and re-selecting NetFlow returns a success message, but flows do not return. Restarting or rebooting the collector appliance does not restore flow collection either.

The collector log records the following message for the affected vCenter on every collection cycle:

User has not selected IPFIX config and there is nothing to manage.

To confirm this applies to a specific vCenter, search the collector logs at /var/log/arkin/collector/collector.STDOUT-*.log for that message. The entry is written by the vcenter.tasks.ManageVCIPFIXTask class, and the thread name identifies which data source it refers to, in the form VCENTER_<vcenter fully qualified domain name>_Config_OpMgr-<number>. A sample entry:

INFO collector [netw@4413 class="vcenter.tasks.ManageVCIPFIXTask" thread="VCENTER_<vcenter fqdn>_Config_OpMgr-#" method="manage" line="267"] User has not selected IPFIX config and there is nothing to manage.

The message is written once per collection cycle for every vCenter data source that has no flow collection selected, so confirm the thread name matches the data source under investigation. Its presence for a data source that has never had flow collection enabled is expected behavior.

IP Flow Information Export (IPFIX) capture files on the collector contain zero records for the affected vCenter across the full retention period, with no gaps in the file sequence.

Environment

VMware VCF Operations for Networks 9.0.2+
VMware vCenter Server data sources with flow collection from vSphere Distributed Switches (VDS)

Cause

The update operation for an existing vCenter data source does not apply the flow collection setting. Per the product Application Programming Interface (API) reference, the update operation for a vCenter data source applies only the nickname, notes and credentials. Flow collection is applied by the add operation.

Once flow collection is cleared on a vCenter data source, re-selecting NetFlow on that existing data source has no effect on the collector. The collector never receives an instruction to manage IPFIX on the distributed switches, so no flow records are generated or sent.

Resolution

Remove the affected vCenter data source and add it again, selecting flow collection during the add operation.

Navigate to Settings > Accounts and Data Sources.
Record the existing configuration of the affected vCenter data source, including the assigned collector, nickname and notes.
Delete the affected vCenter data source.
Select Add Source and add the vCenter again with the same collector and credentials.
During the add, enable NetFlow (IPFIX) and select the distributed switches that require flow collection.
Save the data source.

The collector begins managing IPFIX on the selected distributed switches within one to two minutes, and flow records start arriving in the same minute. Confirm this on the Data Sources page, where the Flows column populates for the re-added source.

For a data source managed through VMware VCF Operations, apply the change through the managing product rather than manually, so that the definition is not overwritten on the next synchronization.

Provide the following information when opening a support request with Broadcom for this issue:

The date and time, in Coordinated Universal Time (UTC), that flow data was last present and when it stopped.
A VCF Operations for Networks support bundle with all Platform virtual machines and all Collector virtual machines selected, collected within 24 hours of the flow data going missing and before any remove and re-add is performed.
A VCF Operations support bundle, where the data sources are managed through that product.
A screenshot of the Data Sources page showing the affected data source.
Details of any configuration change, upgrade or certificate replacement that coincides with the loss of flow data.

Create a VCF Operations for Networks support bundle

On the Settings page, click Infrastructure and Support > Support.
Select the platform virtual machines and the collector virtual machines to include. To select all, click the check box in the header of both the platform and collector tables.
Click Create Support Bundle, then click Yes to confirm.
Click the Download link next to each virtual machine once the bundle is created, and attach the files to the support request.

Bundle creation on a medium sized system takes in excess of fifteen minutes. Only two support bundles exist at one time, so creating a new bundle deletes the oldest existing one. Download any bundle that is still needed before creating another.

For full details, see Create a Support Bundle.

Create a VCF Operations support bundle

Log in to the VCF Operations user interface.
Navigate to Operate > Administration > Control Panel.
Click Support Bundles.
Select the bundle type, either light or full, and select the cluster nodes to include.
Confirm creation, then use the toolbar to download the bundle. Credentials are requested at download.

Generation time depends on the size of the logs and the number of nodes selected. For full details, see Generate VCF Operations Support Bundle in VCF 9.

If the error persists after following these steps, contact Broadcom Support for further assistance.

Additional Information

A remove and re-add resolves the symptom but overwrites the state needed to identify what cleared the flow collection setting. Where the issue recurs, collect the support bundles listed above before performing the remove and re-add.

Related references: