Tanzu Hub dashboards show Error Loading Data due to ensemble_observability_store pod OOMKilled

Products

• VMware Tanzu Application Platform

Issue/Introduction

• Tanzu Hub dashboards display "Error Loading Data".
• The ensemble-observability-store pod in the tanzusm namespace is in CrashLoopBackOff status.
• Describing the pod reveals Reason: OOMKilled and Exit Code: 137.
• During upgrades (e.g., to v10.3.5), logs may incorrectly show patching node failed: nodes not found step="Set Clickhouse nodes labels".

Environment

• VMware Tanzu Hub 10.3.x (including 10.3.7)
• Environments with a high number of applications or resources (e.g., 3000+ apps).
• Hub deployed using the 'evaluation' sizing profile.

Cause

The ensemble_observability_store pod encounters resource exhaustion (Memory/CPU) when processing a large volume of data or handling backlogs after downtime. The 'evaluation' sizing profile does not provide sufficient resources for production-scale data loads.

Resolution

For long-term stability, re-deploy or configure the Hub with a larger sizing profile. Increasing the deployment VM Types, will also increase the resource limits. See Tanzu Hub Sizing Guidelines.

In the meantime, to temporary resolve the issue, you must temporarily allocate enough resources to the ensemble_observability_store pod so it can process the accumulated backlog of requests.

Step 1: Temporarily Increase Pod Resources Manually edit the deployment for the ensemble_observability_store pod to increase its resource limits (CPU and Memory) to match the larger 'foundation' profile sizing.

• Pause the Packageinstall reconciliation 
  

  $ kubectl patch packageinstall/sm  -n tanzusm -p '{"spec":{"paused":true}}' --type=merge
  $ kubectl patch packageinstall/ensemble-helm  -n tanzusm -p '{"spec":{"paused":true}}' --type=merge

   

• Update resources in ensemble_observability_store deployment
  

  $ kubectl edit deployment ensemble-observability-store -n tanzusm

   

Step 2: Allow the Backlog to Clear Monitor the ensemble_observability_store and ClickHouse pods. Once the resources are increased, both pods should transition to a Running state and successfully serve requests. Wait for the burst of queued requests from the downtime to be fully processed.

Step 3: Revert to the Original Profile Once the environment has settled and normal operations resume, manually scale the ensemble_observability_store pod resources back down to the 'evaluation' profile limits.

• Unpause the Packageinstall reconciliation
  

  $ kubectl patch packageinstall/sm  -n tanzusm -p '{"spec":{"paused":false}}' --type=merge
  $ kubectl patch packageinstall/ensemble-helm  -n tanzusm -p '{"spec":{"paused":false}}' --type=merge