Tanzu Hub Dashboards were not loading following an issue with Postgres downtime.

Products

• VMware Tanzu Platform - Hub
• VMware Tanzu for Postgres

Issue/Introduction

If the Postgres DB in Tanzu Hub goes down for any reason, you might encounter an issue where Dashboards fail to load in the Hub UI, even after the database has recovered.

Cause

All Kubernetes jobs will fail during database unavailability period and enter a "Back-off limit exceeded" state. Those jobs initialize various Hub subsystems, including the LeMans ingestion pipelines. From this perspective, the dashboard showing 'no data' is more or less expected.

Resolution

To recover, you must delete the jobs, allow Carvel to recreate them, and a fresh run will occur. 

kubectl get jobs -n tanzusmkubectl delete jobs -n tanzusm $(kubectl get jobs -n tanzusm -o jsonpath='{.items[?(@.status.failed > 0)].metadata.name}' | tr ' ' '\n' | grep -v "history-table-partition-maintenance")

Wait 15 to 20 minutes for reconciliation, then check the jobs again. 

kubectl get jobs -n tanzusm

Once the job runs, you can check the Dashboards from the Hub UI.