Error: Grid processing is running behind after cluster expansion in VCF Operations for Networks

Products

• VCF Operations for Networks

Issue/Introduction

VCF Operations for Networks System GUI Infrastructure and Support page displays the following error: "Grid processing is running behind".

• A significant backlog in flow data processing (e.g., up to 18 hours).
• Data is successfully flowing into the system from collectors, but processing lag increases or remains static.
• System Health may be reported as Poor or Degraded.
• After increasing from 1 Platform node to 3 Platform nodes in the VCF Operations for Networks GUI you may see the following error:

Processing Grid

Realtime processing has been stopped due to a high system load.  Some updates may be delayed

Environment

Product: VCF Operations for Networks 6.14.x (VCF Operations for Networks)
Configuration: Multi-node Platform Cluster (typically observed during or after an expansion from 3 to 5 nodes).

Cause

Following a cluster expansion, the number of Kafka partitions increases (e.g., to 30 partitions). If the flinkjobs service is masked or inactive on the newly added Platform Nodes (e.g., Nodes 4 and 5), the partitions assigned to those specific nodes are not processed. This lack of processing capacity causes a cumulative data backlog and triggers the system alert.

Resolution

To resolve the processing lag, unmask and start the Flink service on platform cluster nodes.

1. SSH to Platform node 1 as the support user.
2. Switch to the ubuntu user:

   ub

3. Check the status of the Flink service to confirm it is masked or inactive (If running and active, no further action required on this node):

   ./run_all/sh sudo systemctl status flinkjobs.service

4. Unmask the Flink service:

   ./run_all/sh sudo systemctl unmask flinkjobs.service

5. Start the Flink service:

   ./run_all/sh sudo systemctl start flinkjobs.service

6. Verify the service is now active and running:

   ./run_all/sh sudo systemctl status flinkjobs.service

7. Monitor the System Health dashboard in the UI. The processing lag should show a steady reduction as the additional nodes begin processing the backlog.

If the service fails to start or the lag does not decrease after 2 hours, collect an updated support bundles for all Platform nodes and contact Broadcom Support.