Service Health Failures and FDB Repartitioning in VCF Operations for Networks Clusters

Products

• VCF Operations for Networks

Issue/Introduction

In a 15-node VCF Operations for Networks cluster, the following symptoms occur simultaneously:

• One or more platform nodes are unreachable via SSH (No route to host).
• The GUI reports One or more essential services are not running or not healthy.
• Error: TSDB Server failed to flush data to HBase or Launcher Service Not Running.
• Alert: Data Retention (Metric Store Maintenance) service is unhealthy on Platform Node 1.
• FoundationDB (FDB) status shows Healthy (Repartitioning) with several GBs of Moving Data.

 

Environment

VCF Operations for Networks

Cause

A node-level hang disrupts the cluster’s distributed data pipeline. Upon recovery via hard reset, FoundationDB (FDB) initiates a high-priority repartitioning process. This intensive activity causes secondary GUI synchronization issues and prevents automated maintenance cron jobs (like Metric Store Retention) from completing on schedule.

Resolution

To restore service health immediately, follow these procedural steps:

1. Identify the unresponsive node using ./run_all.sh uptime from a healthy peer node.
2. Perform a Hard Reset via vCenter if the node does not respond to guest OS commands.
3. Verify SSH access and run ./check-service-health.sh -p -d on all nodes once the affected node is online.
4. If the "Data Retention" alert persists on Node 1, manually execute the metric store maintenance cron job to process the backlog.
5. Monitor repartitioning progress using fdbcli --exec "status detail". Ensure Moving Data reaches 0 GB before performing further reboots or maintenance.
6. If errors are still reported on the GUI, ensure you DO NOT perform any manual shutdown or reboot procedure of Platform Node(s). Contact support by opening a support case with Broadcom Support using the directions at KB 142884 - Creating and managing Broadcom cases to review your VCF Operations for Networks deployment. 

Additional Information

1. For defects and enhancements, subscribe to this article to receive status updates. See KB: 275360 for Subscribe instructions.