Database backup failed for component in VCF Identity Broker or VCF Automation
search cancel

Database backup failed for component in VCF Identity Broker or VCF Automation

book

Article ID: 443584

calendar_today

Updated On:

Products

VCF Automation VCF Operations

Issue/Introduction

  • After upgrading to VMware Cloud Foundation (VCF) 9.1, database backups for the Identity Broker (VIDB) component fail with the error: “Database backup could not be completed for component vidb; please retry the operation or contact support if the issue continues.”

  • Users are unable to log into any VCF component using VCF SSO. The login attempts fail globally with the error message: general.login.error.

  • The "Configure VCF SSO" page appears completely blank when navigating to Manage > Identity & Access > VCF SSO Overview in VMware Cloud Foundation Operations

  • Fleet Lifecycle service pod logs shows below error: 

    /services-logs/vcf-fleet-lcm/###-otg-#####-4xb7d/vcf-fleet-build-service-fleetbuild-#######-#####/fleetbuild/console-log-####.log

    YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR Fleet Build 55 [lcm@4413 scPath="ops" thread="system-task-worker-24" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="appendRefCode:L"] [, taskId=#####-#####-####-####]- (EC399588) (com.broadcom.vcf.lcm.SddcLcmClientTaskFailed) Task with ID '#####-#####-####-####' from SDDC lifecycle with ID '#######-####-####-###-########' has completed with failure(s). Check the errors in the next sub-task(s) for more details.

    YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS INFO Fleet Build 55 [lcm@4413 scPath="ops" thread="system-task-worker-24" logger="c.n.c.c.execution.WorkflowExecutor" method="decide:L"] [] - Execution terminated of workflow: BACKUP_AND_RESTORE_WORKFLOW.1/######-e71f-####-83eb-#######.RUNNING

  • Based on the SDDC Lifecycle logs, it was observed that the backup operation is failing for the VIDB database.

    /services-logs/vcf-sddc-lcm/###-otg-####-4xb7d/vcf-sddc-build-service-sddcbuild-#######-####/sddcbuild/console-log-######.log

    YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR sddc-build-service 54 [lcm@4413 scPath="ops" thread="system-task-worker-17" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="logTaskMessage:96"] [#####-#####-####-####, taskId=#####-#####-####-####] - Database backup failed for component vidb. Please retry the operation or contact support if the issue persists.

    YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR sddc-build-service 54 [lcm@4413 scPath="ops" thread="system-task-worker-17" logger="c.b.v.l.t.o.s.step.StepMessageHelper" method="appendRefCode:125"] [#####-#####-####-####, taskId=#####-#####-####-####] - (########) (com.broadcom.vcf.lcm.VspTaskMessage) Backup workflow failed for component: vidb

  • Identity Broker service pod logs shows below error :

    /services-logs/vidb-external/##-otg-####-2scns/pg-backup-vidb-postgres-instance-backup-####-##-##-##-##-#####/backup-container/console-log-########.log

    YYYY-MM-DDTHH:MM:SS stdout F Executing on-demand WAL backup command in target pod ...
    YYYY-MM-DDTHH:MM:SS stdout F Master pod identified as vidb-postgres-instance-1 is Running
    YYYY-MM-DDTHH:MM:SS stdout F Backup output: curl: (60) SSL: no alternative certificate subject name matches target hostname 'pgbackup-vidb-postgres-instance.vidb-external.svc.cluster.local'

  • VCF operations UI access via SSO failed and the browser shows HTTP error 500 and "<vIDB FQDN> can't currently handle this request."
  • Start component identity broker task status shows "Failed" with an error message "Start of VCF services runtime component '######################' has failed. Check SDDC lifecycle logs for details."

We see the following error while performed pre-check on VCF Automation:

  • Error Code: VCFMS-HEALTH-002
  • Stalled Component: vcd-migrator PackageDeployment (status: Progressing)
  • Observations:
    • kubectl describe pd -n vcd-migrator vcd-migrator shows Phase: Progressing, Ready: Unknown.
    • Logs from vcd-migrator pod indicate an expired 'Trust Root' certificate (expired <MM-DD-YYY>), which appears to be disrupting VMSP reconciliation.
    • No OutOfPods errors found.
    • Logging operator pods are healthy.
    • running the command "kubectl logs pg-backup-vcfapostgres-backup-2026-09-17-09-03-23-zkg79 -n prelude" shows below event:
      Executing on-demand WAL backup command in target pod ... Master pod identified as vcfapostgres-0 is Running Backup output: curl: (56) OpenSSL SSL_read: OpenSSL/3.5.7: error:0A000413:SSL routines::ssl/tls alert unsupported certificate, errno 0 Backup script failed with exit status 56

Environment

  • VCF Identity Broker 9.1
  • VCF Automation 9.1

Cause

The failure is caused by a missing Subject Alternative Name (SAN) entry in the Postgres certificate used by VIDB (pgbackup-vidb-postgres-instance.vidb-external.svc.cluster.local) or VCFA (pgbackup-vcfa-postgres-instance.prelude.svc.cluster.local).

Resolution

  1. A code fix will be made available in the future release. Subscribe to the Knowledge Base Article for updates.
  2. As a workaround, regenerate the PostgreSQL certificate secret to include the missing Subject Alternative Name (SAN) entry.

Regenerate Certificate on VCF Identity Broker:

Regenerate Certificate on VCF Automation:

Perform the following sequence to rotate the certificate in VCF Identity Broker:

  1. Establish an SSH connection to a VCF Services Runtime instance. Refer to the documentation for Enabling SSH Access to the VCF Services Runtime Cluster.
  2. Rotate and Verify Certificate.
     Perform the following sequence to rotate the certificate:
    1. Run the following command to capture the existing certificate request details:
      kubectl get certificaterequest -n vidb-external -o yaml
    2. Delete Existing Secret Trigger automatic regeneration by deleting the current PostgreSQL certificate secret:
      kubectl delete secret vidb-postgres-instance-postgres-cert-secret -n vidb-external
      Note : If step 2 fails, delete any restored certificate requests with the following command:
      kubectl delete certificaterequest -n vidb-external -l velero.io/restore-name=vidb-restore-phase6
    3.  Allow approximately 5 minutes for the automatic regeneration process to complete, during which the system will recreate the secret and rotate the certificate.
    4. Verify SAN Entry Run the following command to inspect the new certificate: 
      kubectl exec -c postgres vidb-postgres-instance-0 -n vidb-external -- cat /tls/tls.crt | openssl x509 -noout -text
    5. Confirm Subject Alternative Names Ensure that pgbackup-vidb-postgres-instance.vidb-external.svc.cluster.local is now present in the Subject Alternative Names (SANs) section of the output.
    6. Retry Backup Once verified, retry the backup operation from the VCF interface.

Perform the following sequence to rotate the certificate in VCF Automation:

  1. Establish an SSH connection to a VCF Services Runtime instance. Refer to the documentation for Enabling SSH Access to the VCF Services Runtime Cluster.
  2. Rotate and verify certificate.
    1. Run the following command to capture the existing certificate request details:
      kubectl get certificaterequest -n prelude -o yaml
    2. Delete Existing Secret Trigger automatic regeneration by deleting the current PostgreSQL certificate secret:
      kubectl delete secret vcfa-postgres-instance-postgres-cert-secret -n prelude
      Note : If step 2 fails/hangs, delete any stale certificate requests with the following command:
      kubectl delete certificaterequest -n prelude -l velero.io/restore-name=vcfa-restore-phase6
    3. Verifiy by running the command:
      kubectl get certificaterequest -n prelude -l velero.io/restore-name=vcfa-restore-phase6
    4.  Allow approximately 5 minutes for the automatic regeneration process to complete, during which the system will recreate the secret and rotate the certificate.
    5. Verify SAN Entry Run the following command to inspect the new certificate: 
      kubectl exec -c postgres vcfa-postgres-instance-0 -n prelude -- cat /tls/tls.crt | openssl x509 -noout -text
    6. Confirm Subject Alternative Names Ensure that 'pgbackup-vcfa-postgres-instance.prelude.svc.cluster.local' is now present in the Subject Alternative Names (SANs) section of the output.
    7. Retry Backup Once verified, retry the backup operation from the VCF interface.

Additional Information

Subscribe to be notified of updates to this Broadcom knowledge article

Powered by Wolken Software