VCF Operations for Networks shows high Processing, Indexer Lag and High FDB moving data

Products

• VCF Operations for Networks

Issue/Introduction

• VCF Operations for Networks shows high Processing, Indexer Lag and High FDB moving data
  
  
• On VCF Operations for Networks GUI, under Settings>Infrastructure and support>Infrastructure  and updates pages you will see High Indexer Lag.
  
  Refer to screenshot below:
  
  
  
  
• Foundation DB service appears to be Running and healthy, when checked, however, FDB moving data is very high raining between 700 GB to 1 TB
  
  You can identify the Foundation DB service and checking moving data by executing below commands on Platfrom1 Node of the Cluster 
   

  ub
  fdbcli
  fdb> status details
  

• Output of Foundation DB replication and moving data as below:
  
  

   Replication health     - Healthy (Repartitioning)
   Moving data            - 750.039 GB
  
     

Environment

VCF Operations for Networks 6.13.0
VCF Operations for Networks 6.14.0
VCF Operations for Networks 6.14.1

Cause

The flow store warmer was aggressively caching, which prevented FDB from processing write requests from the flow pipeline. Consequently, FDB became unhealthy with a high volume of moving data.

FDB lacked sufficient memory to process the data quickly. Increasing the memory allocated to FDB helped reduce the amount of moving data.

Resolution

A fix for this issue would be available in a future version of VCF9.

If you have encountered this issue on VCF Operations for Networks version 6.13, 6.14.0 and 6.14.1, ensure you DO NOT perform any manual shutdown or reboot procedure of Platform Node(s).

Your deployment needs to be evaluated and needs to make in-place database and fdb warmer and heart beat configuration properties changes.

Open a support case with Broadcom Support to review your Aria Operations for Networks deployment. For more information, see Creating and managing Broadcom support cases.