Tanzu Hub installation fails, chi-clickhouse-metrics pod not created

Products

• VMware Tanzu Platform - Hub

Issue/Introduction

Tanzu hub installation errand fails and ensemble-observability-store pod is failing to start because it can connect to clickhouse-metrics service.

Failed to connect to 'service-clickhouse-metrics': Connect to http://service-clickhouse-metrics:8123 failed: Connection refused
2026-01-16T11:17:01,370 WARN  [Thread-2] c.c.c.a.Client <> : Retrying. com.clickhouse.client.api.ClientException: Failed to connect
:
Caused by: org.apache.hc.client5.http.HttpHostConnectException: Connect to http://service-clickhouse-metrics:8123 failed: Connection refused

 

The clickhouse-metrics package is reconciled and clickhouse-metrics chi object is completed. However the chi-clickhouse-metrics-default-0-0-0 pod is not running and there does not appear to be any attempt to start it, no events listed for the pod.

The kube-apiserver logs show that the pod configuration is rejected because the resource request value is greater than the resource limit value.

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod \"chi-clickhouse-metrics-default-0-0-0\" is invalid: spec.containers[0].resources.requests: Invalid value: \"4Gi\": must be less than or equal to memory limit of 3197700k","reason":"Invalid","details":{"name":"chi-clickhouse-metrics-default-0-0-0","kind":"Pod","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"4Gi\": must be less than or equal to memory limit of 3197700k","field":"spec.containers[0].resources.requests"}

Environment

Tanzu Hub 10.3.3 

Cause

The Metrics Store VM is configured with the spec from Evaluation Resource Configuration but other VMs (Control) have been configured with spec from a Small Resource Configuration. This can lead to a situation where Pod resource request values are greater than Pod resource limit values.

Resolution

Ensure specs of all Tanzu Hub VMs match a particular Resource Configuration as outlined in https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-hub/10-3/tnz-hub/install-planning.html