PGO restore in vSphere stack spins and doesn't end

Products

• VMware Tanzu PostgreSQL

Issue/Introduction

Customer will see a backup restore fail in their production stack.

Pods will complete, but the monitor and restored Postgres continue to loop forever in a failed state of trying to restore from a backup.

 

Environment

PGO 2.3

vSphere

Cause

If the kube-janitor is running, it will clear up the "completed" process as though it were something idle, which prevents the process from completing.  

 

Resolution

The kube-janitor cleans up unused resources and, if the resource age (time-to-live) is too low, it will see the "completed" resource as something to remove, even if the window is brief.

Increasing time-to-live values sufficiently will allow the system enough time to complete the restore without the kube-janitor terminating the process prematurely.