Dashboards fail to load after Tanzu Hub upgrade to 10.4.1

Products

• VMware Tanzu Platform Core

Issue/Introduction

Dashboards fail to work after upgrading TanzuHub from 10.3.6 to 10.4.1. The ensemble-observability-store pod logs show errors related to metrics database ingestion. Data is not being ingested, and dashboards appear empty or non-functional.

Cause

The upgrade process creates a backup of the Clickhouse database. During the transition to version 10.4.1, both the live database and the backup database exist simultaneously. The new 10.4 code utilizes an updated schema with stored procedures that expect the new structure. However, the presence of the old schema in the backup database causes conflicts, preventing data ingestion.

Resolution

Perform these steps in the Clickhouse database to remove the conflicting backup views and restore data ingestion:

Review these commands before running them.

DROP VIEW IF EXISTS cdb_hc_b k_bak.samples_agg_hourly_mv ON CLUSTER default;
DROP VIEW IF EXISTS cdb_hc_b k_bak.samples_agg_daily_mv ON CLUSTER default;

Use this command to remove the backup copy of the database:

DROP DATABASE IF EXISTS cdb_hc_bk_bak ON CLUSTER 'default' 

After removing the views, restart the collector VM to resume ingestion:

bosh start hub-tas-collector-####

Confirm dashboards populate correctly once ingestion resumes.

Additional Information

Fix Status

Recommend subscribing to this article to be updated on fix status. Refer to How to subscribe to a Knowledge Article for instructions.