Identity Broker service fails to start in VCF Operations with error "Start of VCF services runtime component has failed"

Products

• VCF Operations

Issue/Introduction

• In VCF Operations, navigating to Build → Lifecycle → VCF Management → Components → Identity Broker shows the service as stopped. Attempting to manually start the service fails with the following error message: Start of VCF services runtime component '########-####-####-############' has failed. Check SDDC lifecycle logs for details.
• Running kubectl get pods -A from one of the VCF Service Runtime (VSP) control plane nodes show the vidb-service pods in a CrashLoopBackOff state.
• Run kubectl describe pod vidb-service-########### -n vidb-external and look at the "Events" section at the bottom to check for issues related to certificates:
  

  ERROR: Exception Primary Class: CertificateExpiredException
  ERROR: Exception Primary Reason: NotAfter: Day Month Date hh:mm:ss UTC Year
  Unexpected error running Liquibase: Connection could not be created to jdbc:postgresql://vidb-postgres-instance.vidb-external.svc:####/vcdb?ssl=true&sslmode=verify-full&sslrootcert=/opt/vmware/idm/certs/bundle.pem...
  SSL error: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

Environment

• VCF Operations 9.1.0.x

Cause

The Identity Broker service experiences an outage because the vidb-service pods fail to initialize. The issue occurs due to an expired internal PostgreSQL SSL certificate, which prevents the required database connection.

Resolution

To resolve this issue, follow the steps mentioned below:

1. Capture the existing certificate request details by running the following command: 

   kubectl get certificaterequest -n vidb-external -o yaml

2. Trigger automatic regeneration by deleting the current PostgreSQL certificate secret:

   kubectl delete secret vidb-postgres-instance-postgres-cert-secret -n vidb-external

3. Delete the associated certificate request:

   kubectl delete certificaterequest -n vidb-external -l velero.io/restore-name=vidb-restore-phase6

4. Delete the failing vidb-service pods to force the Kubernetes cluster to recreate them and mount the freshly generated certificate bundle:

   kubectl delete pod <failing_pod_name> -n vidb-external

5. Watch the new pods to see if they successfully bypass the Init state and proceed to Running:

   kubectl get pods -n vidb-external -w | grep vidb-service

6. In the VCF Operations UI, navigate to Build → Lifecycle → VCF Management → Components →  Identity Broker → Actions → Start Service to try to start the service. 

7. If the service fails to start with the same error message but SSO login starts working, proceed to the next step.

8. Add a suspended label to the VIDB Postgres instance by running the following command:

   kubectl label postgresinstance vidb-postgres-instance -n vidb-external database.vmsp.vmware.com/suspended=true

9. Again, navigate to Build → Lifecycle → VCF Management → Components →  Identity Broker → Actions → Start Service to successfully start the service.