Backup or Upgrade fails due to Stale or invalid Postgres certificate in VCF Automation

Products

VCF Automation

Issue/Introduction

Reviewing the 'Perform Component Upgrade' subtask shows "Database backup failed for component vcfa. Please retry the operation or contact support if the issue persists."
Reviewing running pods on VCF Automation appliance on prelude namespace 'kubectl -n prelude get pods' shows a failed pod called "vcfa-upgrade-91-patch-pre-postgres-cert-clientauth-#####-execute-script-#########" reviewing the logs ('kubectl -n prelude logs vcfa-upgrade-91-patch-pre-postgres-cert-clientauth-#####-execute-script-#########') shows:
```
Waiting for cert-manager to regenerate secret vcfapostgres-postgres-cert-secret after adding 'client auth' to vcfapostgres-postgres-cert (resourceVersion=########, expecting it to change from ########)
```

Describe the certificate using the following command:

kubectl -n prelude describe certificate vcfapostgres-postgres-cert

shows the following Event:

Type Reason Age From Message
---- ------ ---- ---- -------
Warning RequestFailed 3m8s (x2605 over 21h) cert-manager-certificates-request-manager Failed to create CertificateRequest: certificaterequests.cert-manager.io "vcfapostgres-postgres-cert-2" already exists

This error can be observed during VCF Automation 9.x ->9.1.1 upgrade or when the certificates are rotated. Additionally, this error can also be observed sometimes when the VCF Automation is powered off and powered on.
Checking the status of 'CertificateRequests' using the below command displays duplicate certificate requests:
```
kubectl get certificaterequests -n prelude
```

Following symptoms are also observed while performing pre-check on VCF Automation:

Error Code: VCFMS-HEALTH-002
Stalled Component: vcd-migrator PackageDeployment (status: Progressing)

Observations:

1. kubectl describe pd -n vcd-migrator vcd-migrator shows Phase: Progressing, Ready: Unknown.
2. Logs from vcd-migrator pod indicate an expired 'Trust Root' certificate (expired <MM-DD-YYY>), which appears to be disrupting VMSP reconciliation.
3. No OutOfPods errors found.
4. Logging operator pods are healthy.
5. running the command "kubectl logs pg-backup-vcfapostgres-backup-2026-09-17-09-03-23-<ID> -n prelude" shows below event:
```
Executing on-demand WAL backup command in target pod ... Master pod identified as vcfapostgres-0 is Running Backup output: curl: (56) OpenSSL SSL_read: OpenSSL/3.5.7: error:0A000413:SSL routines::ssl/tls alert unsupported certificate, errno 0 Backup script failed with exit status 56
```

Environment

VCF Automation 9.x

Cause

During the VCF Automation upgrade to version 9.1.1 or Power Cycle of VCF Automation, old certificate requests are carried over from version 9.x. Because these old requests contain outdated keys, the system gets stuck and cannot issue new certificates until you delete the old requests.

Resolution

The issue is resolved in the future release. Subscribe to the KB for updates.

Perform the following sequence to rotate the certificate in VCF Automation:

Workaround:

Establish an SSH connection to a VCF Services Runtime instance. Refer to the documentation for Enabling SSH Access to the VCF Services Runtime Cluster.
Rotate and verify certificate.
Run the following command to capture the existing certificate request details:
```
kubectl get certificaterequest -n prelude -o yaml
```
Delete Existing Secret Trigger automatic regeneration by deleting the current PostgreSQL certificate secret:
```
kubectl delete secret vcfa-postgres-instance-postgres-cert-secret -n prelude
```
Note : If step 2 fails/hangs, delete any stale certificate requests with the following command:
```
kubectl delete certificaterequest -n prelude -l velero.io/restore-name=vcfa-restore-phase6
```

Verify by running the command:

kubectl get certificaterequest -n prelude -l velero.io/restore-name=vcfa-restore-phase6

Allow approximately 5 minutes for the automatic regeneration process to complete, during which the system will recreate the secret and rotate the certificate.
```
kubectl get certificaterequests -n prelude
```
Verify that all certificates in the output should show READY = True.

Verify SAN Entry:
Run the following command to inspect the new certificate:

kubectl exec -c postgres vcfa-postgres-instance-0 -n prelude -- cat /tls/tls.crt | openssl x509 -noout -text

Confirm Subject Alternative Names Ensure that 'pgbackup-vcfa-postgres-instance.prelude.svc.cluster.local' is now present in the Subject Alternative Names (SANs) section of the output.
Retry Backup Once verified, retry the backup operation from the VCF interface.

Additional Information

Reference KBs:

Refer to the KB for "Backup or Upgrade fails due to Stale or invalid Postgres certificate in VCF Identity Broker"
Refer to the KB for "Error: "Transport failure reason: TLS_error" in VCF Automation UI"