VCF Automation 9 "prelude" Kubernetes pods in "CrashLoopBackOff", "ImagePullBackOff" or "ErrImagePull" State

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

If you SSH to one of the VCF Automation 9 appliances VCF Automation 9  and run:

kubectl get pods -n prelude -owide | grep -v Running

You may observe "prelude" Kubernetes pods in "CrashLoopBackOff", "ImagePullBackOff", "ErrImagePull" or "Init:Error" state like so:

• The VCFA GUI does not display properly; it might show:
  • No Healthy Upstream
  • HTTP ERROR 404 JSP file [/error.jsp] not found
    
    • GenericJDBCException: could not execute query
• Pods in error status such as tenant-manager may have this FATAL error in their log under /var/log/services-logs:
  • remaining connection slots are reserved for non-replication superuser connections
  • This error is also seen in the vcfapostgres-0 logs, which is where it originates

Environment

VCF Automation 9.0.x

Cause

Snapshot-based backup systems are not supported for use with VCF Automation appliances, resulting in the above-mentioned errors.

These lead to VCF Automation services timing out and continuously being retried, leading to a spike in database connections that hit the "max_connections" limit.

Although the snapshots will likely fail, the attempt to take the snapshots can cause this issue.

For more information, see: Snapshot Management options removed from VCF Automation (VCF-A) and VCF Identity Broker in Fleet Management

Resolution

Disable any snapshot backup solutions or schedules in relation to VCF Automation 9 appliances.

To resolve the crashed pods, reboot the VCFA nodes using VCF Ops Fleet Management.

If the reboot task fails here, contact Broadcom Support for help with manually rebooting the cluster nodes.

Creating and managing Broadcom support cases