Tanzu Hub observability-store post-deploy script errors with "Failed executing restore replica query"

Products

• VMware Tanzu Platform - Hub

Issue/Introduction

Attempting to upgrade Tanzu Hub to 10.4 fails with post-deploy error.

Deploy log shows failure such as:

Task 2295780 | 16:28:05 | Error: Action Failed get_task: Task 94068048-c78a-4cbf-4a1f-426f18fe62c7 result: 1 of 2 post-deploy scripts failed. Failed Jobs: observability-store. Successful Jobs: clickhouse-server.

Error:

[2026-04-15 16:18:04] POST-DEPLOY: Starting observability-store post-deployment tasks...
[2026-04-15 16:18:04] POST-DEPLOY: SSL_ENABLED=false, SERVER_PORT=4010, HTTPS_PORT=4443
[2026-04-15 16:18:04] POST-DEPLOY: Using protocol=http, check_port=4010
[2026-04-15 16:18:04] POST-DEPLOY: Waiting for observability-store to be ready on http://localhost:4010...
[2026-04-15 16:18:04] POST-DEPLOY: Waiting for observability-store... attempt 1/60
...

 2026-04-15T16:27:48,455 WARN  [Thread-6] c.v.a.t.s.u.ClickHouseShardManagerUtils <> : Failed executing restore replica query system restore replica cdb_hc.time_series_v2_telemetry 

This failure indicates potential corruption to clickhouse store data.

Resolution

This problem is caused by corruption of clickhouse database. It's recommended to open a support case to investigate the database corruption.

This issue may be solved by dropping the log store clickhouse store, however this will also result in dataloss of clickhouse metrics and logs.

This can be done by steps:

# bosh -d hub-####### ssh registry

# kubectl -n tanzusm exec -it <click-house-metrics Pod> -- bash

# Retrieve db credentials $ env | grep ADMIN CLICKHOUSE_ADMIN_USER=clickhouse CLICKHOUSE_ADMIN_PASSWORD=##############

# Connect to DB
$ clickhouse-client --user clickhouse --password ########### ClickHouse client version 25.8.7.3 (official build).

Connecting to localhost:9000 as user clickhouse. Connected to ClickHouse server version 25.8.7.

:) show databases; SHOW DATABASES
┌─name───────────────┐
1. │ INFORMATION_SCHEMA │
2. │ cdb_hc │
3. │ default │
4. │ information_schema │
5. │ system │
└────────────────────┘
In order to drop clickhouse DB:

:) drop cdb_hc;

WARNING: This command will cause irreversible loss of clickhouse metrics & data.