Database backup failed for component vidb post VCF 9.1 Upgrade

Products

• VMware Cloud Foundation

Issue/Introduction

• After upgrading to VMware Cloud Foundation (VCF) 9.1, database backups for the Identity Broker (VIDB) component fail with the error: “Database backup could not be completed for component vidb; please retry the operation or contact support if the issue continues.”

  

• Users are unable to log into any VCF component using VCF SSO. The login attempts fail globally with the error message: general.login.error.

• The "Configure VCF SSO" page appears completely blank when navigating to Manage > Identity & Access > VCF SSO Overview in VMware Cloud Foundation Operations

• Fleet Lifecycle service pod logs shows below error: 

  /services-logs/vcf-fleet-lcm/###-otg-#####-4xb7d/vcf-fleet-build-service-fleetbuild-#######-#####/fleetbuild/console-log-####.log

  YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR Fleet Build 55 [lcm@4413 scPath="ops" thread="system-task-worker-24" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="appendRefCode:L"] [, taskId=#####-#####-####-####]- (EC399588) (com.broadcom.vcf.lcm.SddcLcmClientTaskFailed) Task with ID '#####-#####-####-####' from SDDC lifecycle with ID '#######-####-####-###-########' has completed with failure(s). Check the errors in the next sub-task(s) for more details.

  YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS INFO Fleet Build 55 [lcm@4413 scPath="ops" thread="system-task-worker-24" logger="c.n.c.c.execution.WorkflowExecutor" method="decide:L"] [] - Execution terminated of workflow: BACKUP_AND_RESTORE_WORKFLOW.1/######-e71f-####-83eb-#######.RUNNING

• Based on the SDDC Lifecycle logs, it was observed that the backup operation is failing for the VIDB database.

  /services-logs/vcf-sddc-lcm/###-otg-####-4xb7d/vcf-sddc-build-service-sddcbuild-#######-####/sddcbuild/console-log-######.log

  YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR sddc-build-service 54 [lcm@4413 scPath="ops" thread="system-task-worker-17" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="logTaskMessage:96"] [#####-#####-####-####, taskId=#####-#####-####-####] - Database backup failed for component vidb. Please retry the operation or contact support if the issue persists.

  YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR sddc-build-service 54 [lcm@4413 scPath="ops" thread="system-task-worker-17" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="appendRefCode:125"] [#####-#####-####-####, taskId=#####-#####-####-####] - (########) (com.broadcom.vcf.lcm.VspTaskMessage) Backup workflow failed for component: vidb

• Identity Broker service pod logs shows below error :

  /services-logs/vidb-external/##-otg-####-2scns/pg-backup-vidb-postgres-instance-backup-####-##-##-##-##-#####/backup-container/console-log-########.log

  YYYY-MM-DDTHH:MM:SS stdout F Executing on-demand WAL backup command in target pod ...
YYYY-MM-DDTHH:MM:SS stdout F Master pod identified as vidb-postgres-instance-1 is Running
YYYY-MM-DDTHH:MM:SS stdout F Backup output: curl: (60) SSL: no alternative certificate subject name matches target hostname 'pgbackup-vidb-postgres-instance.vidb-external.svc.cluster.local'

 

Environment

• VMware Cloud Foundation (VCF) 9.1
• VCF Operations 9.1
• VCF Identity Broker 9.1

Cause

The failure is caused by a missing Subject Alternative Name (SAN) entry for pgbackup-vidb-postgres-instance.vidb-external.svc.cluster.local in the Postgres certificate used by the Identity Broker.

Resolution

Broadcom Engineering team is aware of this issue, and a fix is planned for an upcoming release.

As a workaround, regenerate the PostgreSQL certificate secret to include the missing Subject Alternative Name (SAN) entry.

1. Establish an SSH connection to a VCF Services Runtime instance. Refer to the documentation for Enabling SSH Access to the VCF Services Runtime Cluster.
2. Rotate and Verify Certificate.
    Perform the following sequence to rotate the certificate:
   1. Run the following command to capture the existing certificate request details:

      kubectl get certificaterequest -n vidb-external -o yaml

   2. Delete Existing Secret Trigger automatic regeneration by deleting the current PostgreSQL certificate secret:

      kubectl delete secret vidb-postgres-instance-postgres-cert-secret -n vidb-external

      Note : If step 2 fails, delete any restored certificate requests with the following command:

      kubectl delete certificaterequest -n vidb-external -l velero.io/restore-name=vidb-restore-phase6

   3.  Allow approximately 5 minutes for the automatic regeneration process to complete, during which the system will recreate the secret and rotate the certificate.
   4. Verify SAN Entry Run the following command to inspect the new certificate: 

      kubectl exec -c postgres vidb-postgres-instance-0 -n vidb-external -- cat /tls/tls.crt | openssl x509 -noout -text

   5. Confirm Subject Alternative Names Ensure that pgbackup-vidb-postgres-instance.vidb-external.svc.cluster.local is now present in the Subject Alternative Names (SANs) section of the output.
   6. Retry Backup Once verified, retry the backup operation from the VCF interface.

Additional Information

Subscribe to be notified of updates to this Broadcom knowledge article