Daedalus pods periodically restarting with ReplicaSet updates - Tanzu Hub

Products

• VMware Tanzu Platform - Hub

Issue/Introduction

Daedalus pods in the tanzusm namespace appear to restart or start periodically without the container restart counter incrementing. This behavior is typically observed when the service hits a database connection timeout or a reconciliation deadline. 

• daedalus pods start periodically, but the ReplicaSet ID remains unchanged.
• The container restart counter does not increment.
• ReplicaSet events show frequent updates or ProgressDeadlineExceeded.
• Logs show: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection or request timed out after 30000ms.
• Database health contributor response times exceed 20,000ms.

Environment

• VMware Tanzu Platform - Hub 10.4
• VMware Tanzu Platform - Hub 10.4.1
•  

Cause

This issue is caused by a combination of factors:

1. OOM Condition: Large environments may hit resource limits.
2. Database Timeout: The daedalus pod fails to connect to the PostgreSQL database within the default 30-second window.
3. Proxy Failure: In 10.4, Trivy pods may fail to utilize proxy configurations, leading to reconcile timeouts.

Resolution

Fixed in Tanzu Hub 10.4.2 and higher. See Download Broadcom products and software for steps to download this release.

If you are on version 10.4 or 10.4.1, use the following steps to diagnose:

1. Check for ReplicaSet updates: kubectl describe deployment daedalus -n tanzusm
2. Review logs for JDBC timeouts: kubectl logs -n tanzusm [pod-id] | grep -i timeout
3. If a proxy is in use, follow the workaround in Article 439756 to direct Trivy to a local registry.
4. Subscribe to this article to receive updates on fix status (reference: Article 275360).

Additional Information

• For procedural assistance, contact Broadcom Support.