Timeseries Cache Server Failed to Initialize Correctly and Data Retention (Metric Store Maintenance) service is unhealthy error After HDFS Block Corruption

Products

• VCF Operations for Networks

Issue/Introduction

As a result of a storage outage affecting Operations for Networks platform nodes, administrators observe a "Timeseries cache server failed to initialize correctly" error on one or more platform nodes. A "Data Retention service unhealthy" alarm may also be present on platform nodes.

Environment

VCF Operations for Networks 9.0.2

Cause

The tsdb-server service fails to initialize and remains in a hung state following an underlying HDFS block corruption event and subsequent HBase data inconsistencies.

Resolution

• Identify any corrupt HDFS blocks using the following command:

  • sudo -u hdfs hdfs fsck /

• If the corrupt blocks are small in number (less than 100), delete them by executing:

  • sudo -u hdfs hdfs fsck / -delete

• Restart the HBase master on platform nodes 1-3 and the region servers on all nodes by running the following commands in sequence:

  • stopm

  • stopr

  • startm

  • startr

• Validate the HFiles by executing:

  • hbck -checkCorruptHFiles 2>&1 | grep "HFiles corrupted"

• If the output is non-zero, sideline the corrupted files using the command:

  • hbck -sidelineCorruptHFiles

• Verify cluster health and confirm zero inconsistencies by running:

  • hbck

• If inconsistencies remain, execute the cluster repair script:

  • sudo /home/ubuntu/build-target/hbase/hbase-cluster-repair.sh

• To clear any residual "Timeseries cache server failed to initialize correctly" alarms, log into the affected platform node(s) and restart the service using the command:

  • sudo service tsdb-server restart

If the corrupt blocks are greater than 100, or if the steps above do not resolve all alarms for platform nodes, please open a Support Request with Broadcom to investigate further.