Tanzu Hub Upgrade - Observability Store Pods CrashLoopBackOff due to Liquibase Failures after ClickHouse Shard PV Recreation
Article ID: 449547
Updated On:
Products
Issue/Introduction
Symptoms
- ensemble-helm reconciliation fails during the Tanzu Hub upgrade.
- ensemble-observability-store pod is in CrashLoopBackOff (CLBO) due to liquibase failures
How to Check for the Issue
Step 1: Check the ensemble-helm reconciliation status
The Hub install/upgrade will fail with an error like below:
10:10:46AM: fail: reconcile packageinstall/sm (packaging.carvel.dev/v1alpha1) namespace: tanzusm
10:10:46AM: ^ Reconcile failed: message: kapp: Error: waiting on reconcile packageinstall/ensemble-helm (packaging.carvel.dev/v1alpha1) namespace: tanzusm:
Finished waiting unsuccessfully:
Reconcile failed: message: kapp:
Error:
Timed out waiting after 25m0s for resources: [deployment/ensemble-observability-store (apps/v1) namespace: tanzusm]
[x] Installation failed with error: kapp deploy failed: command execution failed: exit status 1:
[i] Cluster logs collection start...
Step 2: Check the ensemble-observability-store pod logs
kubectl logs -n tanzusm ensemble-observability-store-<PodID>
The pod logs show a liquibase changeset failure because the target ClickHouse database (e.g. cdb_hc) does not exist:
Running Changeset: liquibase/v2/clickhouse-metrics.sql::1::jhwang
2026-07-16T15:22:05,311 ERROR [Thread-0] l.l.c.JavaLogger <> : ChangeSet liquibase/v2/clickhouse-metrics.sql::1::jhwang encountered an exception. liquibase.exception.DatabaseException: Code: 81. DB::Exception: Database cdb_hc does not exist. (UNKNOWN_DATABASE) (version 26.3.15.4 (official build))Code: 81. DB::Exception: There was an error on [chi-clickhouse-metrics-default-1-0-0.service-clickhouse-metrics-default-1-0.tanzusm.svc.cluster.local:9000]: Code: 81. DB::Exception: Database cdb_hc does not exist. (UNKNOWN_DATABASE) (version 26.3.15.4 (official build)). (UNKNOWN_DATABASE) (version 26.3.15.4 (official build)) [Failed SQL: (0) ALTER TABLE cdb_hc.samples_custom ON CLUSTER 'default' MODIFY TTL
toDate(fromUnixTimestamp(timestamp)) + toIntervalHour(168) WHERE __domain__ = 'observability-realtime',
toDate(fromUnixTimestamp(timestamp)) + toIntervalHour(720)
settings materialize_ttl_after_modify=0]
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:497)Environment
Tanzu Hub
Cause
Root cause of the shard-1 PV recreation on upgrade:
- Trigger: the Altinity operator's default host policy onDataLoss: recreate.
- What it did: during a redeploy, shard-1's host tripped the operator's data-loss detector, so the operator deleted and recreated the data-volume-claim-...-1-0-0 PVC. The node-local provisioner made a fresh empty hostpath directory on the same BOSH node (/var/vcap/store/...), and the old PV — held by Retain — dropped to Released. Shard 1 came back empty, so its database/tables (e.g. cdb_hc) no longer existed, which is why the liquibase changesets above failed.
- Why only shard 1: single replica per shard + node-local hostpath. On a BOSH node whose /var/vcap/store momentarily reads as empty (disk reattach/remount during the redeploy), the host looks "data-lost" → recreate wipes it. Shards 0 and 2 didn't trip it. With replicasCount: 1 there's no peer to rebuild from, so the wipe is permanent.
Resolution
1.a Pause the packages clickhouse-metrics, clickhouse-operator and sm
This is required to ensure the operator does not interfere with the restoration.
kctrl package installed pause --package-install clickhouse-metrics -n tanzusm
kctrl package installed pause --package-install clickhouse-operator -n tanzusm
kctrl package installed pause --package-install sm -n tanzusm
1.b Scale down the store pods to 0, to avoid them connecting to CH and messing up with schema
kubectl -n tanzusm scale deployment ensemble-observability-store --replicas=0
2. Scale down the affected clickhouse statefulset to 0
kubectl -n tanzusm scale statefulset chi-clickhouse-metrics-default-1-0 --replicas=0
3. Identify the old PV which was created newly
kubectl -n tanzusm get pv|grep clickhouse
pvc-185ed9bc-64ea-4f86-aadd-############ 12Gi RWO Retain Bound tanzusm/data-volume-claim-chi-clickhouse-metrics-default-0-0-0 tp-storage-class <unset> 27h
pvc-288075c3-2cd3-4c1c-815d-############ 12Gi RWO Retain Bound tanzusm/data-volume-claim-chi-clickhouse-metrics-default-1-0-0 tp-storage-class <unset> 24h
pvc-7c9df018-2ac9-4e83-bee0-############ 12Gi RWO Retain Released tanzusm/data-volume-claim-chi-clickhouse-metrics-default-1-0-0 tp-storage-class <unset> 27h
pvc-7f9b9de6-efb5-474c-9f1f-############# 12Gi RWO Retain Bound tanzusm/data-volume-claim-chi-clickhouse-metrics-default-2-0-0 tp-storage-class <unset> In this case pvc-7c9df018-2ac9-4e83-bee0-############ which is in released state.
4. Free the old PV so it can be rebound
kubectl -n tanzusm patch pv pvc-7c9df018-2ac9-4e83-bee0-############ -p '{"spec":{"claimRef": null}}'Its status should change to Available.
kubectl get pv pvc-7c9df018-2ac9-4e83-bee0-############
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-7c9df018-2ac9-4e83-bee0-############ 12Gi RWO Retain Available tp-storage-class <unset> 27h
5. Backup the current PVC
kubectl -n tanzusm get pvc data-volume-claim-chi-clickhouse-metrics-default-1-0-0 -o yaml > /tmp/wrong-pvc-backup.yaml
6. Delete the existing PVC
kubectl -n tanzusm delete pvc data-volume-claim-chi-clickhouse-metrics-default-1-0-0
7. Based on /tmp/wrong-pvc-backup.yaml and below template populate the new PVC contents and save it to recovered-pvc.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-volume-claim-chi-clickhouse-metrics-default-1-0-0
namespace: tanzusm
labels:
# paste from /tmp/wrong-pvc-backup.yaml
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: <same as old PV, e.g. tp-storage-class>
volumeName: <old-pv-name>
resources:
requests:
storage: <old PV's capacity>
8. Re-create the PVC binding to the old PV
kubectl -n tanzusm apply -f recovered-pvc.yml
9. Bring the pod back, it should bind to the newly created PVC and the old volume
kubectl -n tanzusm scale statefulset chi-clickhouse-metrics-default-1-0 --replicas=1
10. Make sure all clickhouse pods are available and there is data in the shard which was restored
kubectl -n tanzusm get po|grep clickhouse
11. If the data is fine, scale up the store pods back
kubectl -n tanzusm scale deployment ensemble-observability-store --replicas=1
12. Usually since shard-1 now comes up with old keeper metadata we see below DDL failures on it.
Seeing these logs in store as the DDL is failing.
ERROR: Exception Primary Source: 4.31.1
2026-07-17T10:32:11,961 INFO [Thread-0] l.l.c.JavaLogger <> : Command execution complete
2026-07-17T10:32:12,752 INFO [Thread-0] l.l.c.JavaLogger <> : Successfully released change log lock
Successfully released all database change log locks for 'clickhouse@jdbc:clickhouse://chi-clickhouse-metrics-default-0-0-0.service-clickhouse-metrics-default-0-0:8123/default'
2026-07-17T10:32:12,757 INFO [Thread-0] l.l.c.JavaLogger <> : Command execution complete
2026-07-17T10:32:12,759 INFO [Thread-0] c.v.a.t.s.u.SchemaManagerUtils <> : remoteUserWithStandardPasswordEnabled is set to false
2026-07-17T10:32:12,825 INFO [Thread-0] l.l.c.JavaLogger <> : Reading from default.DATABASECHANGELOG
2026-07-17T10:32:13,369 INFO [Thread-0] l.l.c.JavaLogger <> : Successfully acquired change log lock
2026-07-17T10:32:13,370 INFO [Thread-0] l.l.c.JavaLogger <> : Using deploymentId: 4284223135
2026-07-17T10:32:13,378 INFO [Thread-0] l.l.c.JavaLogger <> : Reading from default.DATABASECHANGELOG
Running Changeset: liquibase/v2/clickhouse-metrics.sql::1::jhwang
2026-07-17T10:35:14,579 WARN [Thread-0] c.c.c.a.Client <> : Retrying. com.clickhouse.client.api.ServerException: Code: 159. DB::Exception: Distributed DDL task /clickhouse/task_queue/ddl/query-0000003782 is not finished on 1 of 3 hosts (0 of them are currently executing the task, 0 are inactive). They are going to execute the query in the background. Was waiting for 180.8954398 seconds, which is longer than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED) (version 26.3.15.4 (official build))
at com.clickhouse.client.api.internal.HttpAPIClientHelper.readError(HttpAPIClientHelper.java:403)
at com.clickhouse.client.api.internal.HttpAPIClientHelper.executeRequest(HttpAPIClientHelper.java:467)
at com.clickhouse.client.api.Client.lambda$query$5(Client.java:1599)
at com.clickhouse.client.api.Client.runAsyncOperation(Client.java:2012)
at com.clickhouse.client.api.Client.query(Client.java:1644)
at com.clickhouse.client.api.Client.query(Client.java:1542)
at com.clickhouse.jdbc.StatementImpl.executeUpdateImpl(StatementImpl.java:219)
at com.clickhouse.jdbc.StatementImpl.execute(StatementImpl.java:343)
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:491)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:83)
chi-clickhouse-metrics-default-0-0-0.service-clickhouse-metrics-default-0-0.tanzusm.svc.cluster.local :) create database test on cluster 'default'
CREATE DATABASE test ON CLUSTER default
Query id: 5a116265-ba5f-41b4-aecb-8a9b32e90860
↓ Progress: 2.00 rows, 286.00 B (0.02 rows/s., 2.61 B/s.) 66%
┌─host──────────────────────────────────────────────────────────────────────────────────────────────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
1. │ chi-clickhouse-metrics-default-0-0-0.service-clickhouse-metrics-default-0-0.tanzusm.svc.cluster.local │ 9000 │ 0 │ │ 2 │ 0 │
2. │ chi-clickhouse-metrics-default-2-0-0.service-clickhouse-metrics-default-2-0.tanzusm.svc.cluster.local │ 9000 │ 0 │ │ 1 │ 0 │
└───────────────────────────────────────────────────────────────────────────────────────────────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘
2 rows in set. Elapsed: 180.793 sec.
Received exception from server (version 26.3.15):
Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: Distributed DDL task /clickhouse/task_queue/ddl/query-0000003784 is not finished on 1 of 3 hosts (0 of them are currently executing the task, 0 are inactive). They are going to execute the query in background. Was waiting for 180.799411945 seconds, which is longer than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED)
13. Kick the packages clickhouse-metrics, clickhouse-operator and sm
kctrl package installed kick --package-install clickhouse-metrics -n tanzusm
kctrl package installed kick --package-install clickhouse-operator -n tanzusm
kctrl package installed kick --package-install sm -n tanzusm
Additional Information
Steps to recover from the DDL timeouts on shard-1
1. Run the command below to see the state of keeper on all shards, make sure that the shard-0 is either a follower or leader.
for i in 0 1 2; do echo "=== Node $i ==="; kubectl -n tanzusm exec chi-clickhouse-metrics-default-$i-0-0 -- clickhouse-keeper-client -h 127.0.0.1 -p 2181 -q "stat" --history-file=/dev/null | grep -E "(Mode:|Node count:)" || echo "Failed"; done
2. If this succeeds the read_only or session expired replicas may need to be restored/restarted. Usually this should not be the case as store pod do this.
2.a Check for stuck replicas on shard-1:
kubectl -n tanzusm exec chi-clickhouse-metrics-default-1-0-0 -c clickhouse -- \
clickhouse-client -u default --password="$CLICKHOUSE_ADMIN_PASSWORD" -q \
"SELECT database, table, is_readonly, is_session_expired, zookeeper_exception FROM system.replicas WHERE is_readonly=1 OR is_session_expired=1 FORMAT Vertical"2.b Restart session-expired replicas first, then restore read-only ones (same order the job uses — restarting first re-establishes the ZK/keeper session, which read-only tables need before RESTORE will succeed):
# for each row where is_session_expired=1:
kubectl -n tanzusm exec chi-clickhouse-metrics-default-1-0-0 -c clickhouse -- \
clickhouse-client -u default --password="$CLICKHOUSE_ADMIN_PASSWORD" -q "SYSTEM RESTART REPLICA <db>.<table>"
# wait ~10s for reconnect, then for each remaining row where is_readonly=1 and is_session_expired=0:
kubectl -n tanzusm exec chi-clickhouse-metrics-default-1-0-0 -c clickhouse -- \
clickhouse-client -u default --password="$CLICKHOUSE_ADMIN_PASSWORD" -q "SYSTEM RESTORE REPLICA <db>.<table>"2.c Re-check and retest DDL:
kubectl -n tanzusm exec chi-clickhouse-metrics-default-1-0-0 -c clickhouse -- \
clickhouse-client -u default --password="$CLICKHOUSE_ADMIN_PASSWORD" -q \
"SELECT count() FROM system.replicas WHERE is_readonly=1 OR is_session_expired=1"
# should be 0, then:
kubectl -n tanzusm exec chi-clickhouse-metrics-default-1-0-0 -c clickhouse -- \
clickhouse-client -u default --password="$CLICKHOUSE_ADMIN_PASSWORD" -q \
"CREATE DATABASE IF NOT EXISTS health_check ON CLUSTER 'default'; DROP DATABASE IF EXISTS health_check ON CLUSTER 'default'"
If some tables are still stuck after one pass, repeat steps 2.b-2.c once or twice more (the existing job retries up to 2x with 10-20s backoff — session reconnection isn't always instant).
3. If we see the shard-1 keeper has not joined the cluster we might need to delete the keeper data on it.
kubectl -n tanzusm logs chi-clickhouse-metrics-default-1-0-0 -c clickhouse | grep -iE "raft|keeper|coordination" | tail -50If you see repeated errors (not just "installing snapshot" progress, which is normal and self-resolving), the fix is to wipe only the keeper coordination state on shard-1 (not the table data — separate directory on the same PVC) and let it rejoin as an empty follower, which forces a full fresh snapshot pull from the current leader (shard-0 or shard-2):
kubectl -n tanzusm exec chi-clickhouse-metrics-default-1-0-0 -c clickhouse -- \
rm -rf /bitnami/clickhouse/clickhouse-keeper/coordination/log/* /bitnami/clickhouse/clickhouse-keeper/coordination/snapshots/*
kubectl -n tanzusm delete pod chi-clickhouse-metrics-default-1-0-0 # let the STS recreate it, keeper rejoins freshThen repeat step 1 to confirm the node rejoined, and steps 2 to clear any remaining read-only tables.
Clean up new PV
1. Once the store pods have come up fine and ingestion and queries are working we can clean up the new PV which was created.
Use the below command to find out the new PV and the one in Releases state on shard-1 is the one we should delete. In this case its pvc-288075c3-2cd3-4c1c-815d-############
kubectl -n tanzusm get pv|grep clickhouse
pvc-185ed9bc-64ea-4f86-aadd-############ 12Gi RWO Retain Bound tanzusm/data-volume-claim-chi-clickhouse-metrics-default-0-0-0 tp-storage-class <unset> 3d23h
pvc-288075c3-2cd3-4c1c-815d-############ 12Gi RWO Retain Released tanzusm/data-volume-claim-chi-clickhouse-metrics-default-1-0-0 tp-storage-class <unset> 3d21h
pvc-7c9df018-2ac9-4e83-bee0-############ 12Gi RWO Retain Bound tanzusm/data-volume-claim-chi-clickhouse-metrics-default-1-0-0 tp-storage-class <unset> 3d23h
pvc-7f9b9de6-efb5-474c-9f1f-############ 12Gi RWO Retain Bound tanzusm/data-volume-claim-chi-clickhouse-metrics-default-2-0-0 tp-storage-class <unset> 2. Delete the pvc from above step
kubectl -n tanzusm delete pv pvc-288075c3-2cd3-4c1c-815d-#############
Warning: deleting cluster-scoped resources, not scoped to the provided namespace
persistentvolume "pvc-288075c3-2cd3-4c1c-815d-##############" deletedFeedback
