Backup or Upgrade fails due to Stale or invalid Postgres certificate in VCF Identity Broker

Products

VCF Operations

Issue/Introduction

A problem with the PostgreSQL certificate / stale or duplicate cert-manager CertificateRequests.
database backups for the Identity Broker (VIDB) component fail with the error: “Database backup could not be completed for component vidb; please retry the operation or contact support if the issue continues.”
After upgrading to VMware Cloud Foundation (VCF) 9.1, database backups for the Identity Broker (VIDB) component fail with the error: “Database backup could not be completed for component vidb; please retry the operation or contact support if the issue continues.”
Users are unable to log into any VCF component using VCF SSO. The login attempts fail globally with the error message: general.login.error.
The "Configure VCF SSO" page appears completely blank when navigating to Manage > Identity & Access > VCF SSO Overview in VMware Cloud Foundation Operations.

Fleet Lifecycle service pod logs shows below error:

/services-logs/vcf-fleet-lcm/###-otg-#####-4xb7d/vcf-fleet-build-service-fleetbuild-#######-#####/fleetbuild/console-log-####.log

YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR Fleet Build 55 [lcm@4413 scPath="ops" thread="system-task-worker-24" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="appendRefCode:L"] [, taskId=#####-#####-####-####]- (EC399588) (com.broadcom.vcf.lcm.SddcLcmClientTaskFailed) Task with ID '#####-#####-####-####' from SDDC lifecycle with ID '#######-####-####-###-########' has completed with failure(s). Check the errors in the next sub-task(s) for more details.
YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS INFO Fleet Build 55 [lcm@4413 scPath="ops" thread="system-task-worker-24" logger="c.n.c.c.execution.WorkflowExecutor" method="decide:L"] [] - Execution terminated of workflow: BACKUP_AND_RESTORE_WORKFLOW.1/######-e71f-####-83eb-#######.RUNNING

Based on the SDDC Lifecycle logs, it was observed that the backup operation is failing for the VIDB database.

/services-logs/vcf-sddc-lcm/###-otg-####-4xb7d/vcf-sddc-build-service-sddcbuild-#######-####/sddcbuild/console-log-######.log

YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR sddc-build-service 54 [lcm@4413 scPath="ops" thread="system-task-worker-17" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="logTaskMessage:96"] [#####-#####-####-####, taskId=#####-#####-####-####] - Database backup failed for component vidb. Please retry the operation or contact support if the issue persists.
YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR sddc-build-service 54 [lcm@4413 scPath="ops" thread="system-task-worker-17" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="appendRefCode:125"] [#####-#####-####-####, taskId=#####-#####-####-####] - (########) (com.broadcom.vcf.lcm.VspTaskMessage) Backup workflow failed for component: vidb

Identity Broker service pod logs shows below error :

/services-logs/vidb-external/##-otg-####-2scns/pg-backup-vidb-postgres-instance-backup-####-##-##-##-##-#####/backup-container/console-log-########.log

YYYY-MM-DDTHH:MM:SS stdout F Executing on-demand WAL backup command in target pod ...
YYYY-MM-DDTHH:MM:SS stdout F Master pod identified as vidb-postgres-instance-1 is Running
YYYY-MM-DDTHH:MM:SS stdout F Backup output: curl: (60) SSL: no alternative certificate subject name matches target hostname 'pgbackup-vidb-postgres-instance.vidb-external.svc.cluster.local'

VCF operations UI access via SSO failed and the browser shows HTTP error 500 and "<vIDB FQDN> can't currently handle this request."

Start component identity broker task status shows "Failed" with the following error message:

"Start of VCF services runtime component '######################' has failed. Check SDDC lifecycle logs for details."

Environment

VCF Identity Broker 9.x

Cause

The failure is caused by a missing Subject Alternative Name (SAN) entry in the Postgres certificate used by VIDB (pgbackup-vidb-postgres-instance.vidb-external.svc.cluster.local)

Resolution

A code fix will be made available in the future release. Subscribe to the Knowledge Base Article for updates.
As a workaround, regenerate the PostgreSQL certificate secret to include the missing Subject Alternative Name (SAN) entry.

Regenerate Certificate on VCF Identity Broker:

Perform the following sequence to rotate the certificate in VCF Identity Broker:

Establish an SSH connection to a VCF Services Runtime instance. Refer to the documentation for Enabling SSH Access to the VCF Services Runtime Cluster.
Rotate and Verify Certificate.
Perform the following sequence to rotate the certificate:
1. Run the following command to capture the existing certificate request details:
```
kubectl get certificaterequest -n vidb-external -o yaml
```
2. Delete Existing Secret Trigger automatic regeneration by deleting the current PostgreSQL certificate secret:
```
kubectl delete secret vidb-postgres-instance-postgres-cert-secret -n vidb-external
```
  Note : If step 2 fails, delete any restored certificate requests with the following command:
```
kubectl delete certificaterequest -n vidb-external -l velero.io/restore-name=vidb-restore-phase6
```
3. Allow approximately 5 minutes for the automatic regeneration process to complete, during which the system will recreate the secret and rotate the certificate.
4. Verify SAN Entry Run the following command to inspect the new certificate:
```
kubectl exec -c postgres vidb-postgres-instance-0 -n vidb-external -- cat /tls/tls.crt | openssl x509 -noout -text
```
5. Confirm Subject Alternative Names Ensure that pgbackup-vidb-postgres-instance.vidb-external.svc.cluster.local is now present in the Subject Alternative Names (SANs) section of the output.
6. Retry Backup Once verified, retry the backup operation from the VCF interface.

Additional Information

Reference KBs:

Refer to the KB for "Backup or Upgrade fails due to Stale or invalid Postgres certificate in VCF Automation"
Refer to the KB for "Error: 'Transport failure reason: TLS_error' in VCF Automation UI"