Unable to login to VCF Operations for Networks GUI after cluster reboot due to FoundationDB and HDFS corruption

Products

• VCF Operations for Networks

Issue/Introduction

• VCF Operations for Networks GUI is inaccessible after a cluster reboot.
• ./check-service-health.sh -u -d indicates that database services (HDFS, FoundationDB, or HRegionServer) are in a "Critical," "Unhealthy," or "Unknown" state.
• Running hdfs fsck / shows a high count of "CORRUPT FILES" and "MISSING BLOCKS."
• Running fdbcli and executing status details returns:
  • Replication health - UNHEALTHY: No replicas remain of some data
  • Moving data - unknown
  • Sum of key-value sizes - unknown
  • Deployment: Multi-node platform cluster 

Environment

• VCF Operations for Networks 6.14.x

Cause

Irreparable database corruption occurred within the HDFS and FoundationDB layers. While HDFS corruption can often be cleared, FoundationDB corruption involving a total loss of replicas prevents the metadata layer from initializing, making the deployment unrecoverable.

Resolution

Based on the extent of the corruption in the FoundationDB (FDB) layer, if the replication health shows "No replicas remain," the deployment cannot be recovered through standard service restarts or scripts.

1. Verify Corruption Extent:

   • Log into the Platform 1 node via SSH as support.
   • Check FDB status: fdbcli --exec "status".
   • If the output shows "No replicas remain of some data" and "Fault Tolerance - 0 machines," the database integrity is lost.

2. Attempt HDFS Clear (Optional - Results in Data Loss):

   • If HDFS is the only component corrupted, you can try: hdfs fsck / -delete.
   • Note: This deletes corrupted metric blocks. If UI access is not restored after this, proceed to step 3.

3. Redeploy Cluster:

   • Since the FoundationDB (which holds the cluster's configuration and metadata) is beyond repair, a full redeployment of the VCF Operations for Networks is required.
   • Deploy a fresh instance of the same version (e.g., 6.14.2).
   • Restore from a known good configuration backup if available.