Flink Container Service Flapping due to HBase TableNotFoundException in VCF Operations for Networks
search cancel

Flink Container Service Flapping due to HBase TableNotFoundException in VCF Operations for Networks

book

Article ID: 435156

calendar_today

Updated On:

Products

VCF Operations for Networks VMware vSphere ESXi

Issue/Introduction

Users may experience data processing failures in VCF Operations for Networks with the following symptoms:

  • Critical alerts in the UI stating "Platform Health: Data Source Failed".

  • System Health problems indicating "Grid processing is running behind" and "TSDB server failed to flush data to HBase."

  • Running the CLI command ./check-service-health.sh -p -d shows the FlinkContainer service is flapping (the uptime constantly resets to a few minutes or seconds).

  • The hbase hbck command may report 0 inconsistencies, despite the ongoing issues.

  • Checking the Flink taskmanager.log reveals the following fatal error:

"ERROR hbase.client.AsyncProcess AnomalyScoreComputer -> Sink: AnomalyScoreSink (1/8)_25 submit:283 Failed to get region location
org.apache.hadoop.hbase.TableNotFoundException: metrics.20"

Environment

VCF Operations for Networks

Cause

This issue occurs due to severe corruption or loss of core metric tables (such as metrics.20) within the underlying HBase database.

The Flink application (metricsjob) is responsible for processing metrics and anomaly scores, which it attempts to write into HBase. Because the target table is missing, HBase returns a TableNotFoundException. This fatal error causes the Flink TaskManager to crash. The resource manager (YARN) detects the crash and continuously restarts the container, resulting in the flapping service status.

Note: The hbase hbck tool may report "0 inconsistencies" because it only validates the structural integrity of tables currently registered in the database metadata; it does not detect if a table was completely dropped or lost.

Resolution

Because the core underlying tables are completely missing from the database, standard service restarts and HBase repair scripts (hbck) will not resolve the issue.

To resolve this, choose one of the following paths:

Option 1: Environment Redeployment (Recommended) Due to the extensive manual reconfiguration required and the permanent loss of historical metric data, deploying a fresh instance of the Aria Operations for Networks Platform and Collector nodes is the cleanest and most reliable path forward.

Option 2: Manual Table Recreation via Broadcom Support Broadcom Engineering can manually force the recreation of the missing tables.

  • Warning: This approach will result in the permanent loss of all historical metric data.

  • Warning: This approach requires the administrator to manually re-add and re-validate all Data Sources only on best effort basis.

Additional Information

To manually verify if the tables are missing from the system, an administrator can run the following command from the Platform VM CLI (logged in as the support or ubuntu user): echo "list" | sudo -u hbase hbase shell Review the output to confirm the absence of the metrics.20 table (or other metrics.x tables).