VCF Identity & Access Failure: vidb-service Pods in Init:CrashLoopBackoff due to accesscontrol_configkeystore.bcfks Duplicate Key

Products

• VCF Operations
• VMware Cloud Foundation

Issue/Introduction

• Following a datastore capacity exhaustion event in the management cluster, virtual machines entering a bad state, and subsequent reboots, Identity & Access functionality becomes unavailable for the VCF instance. The VIDB fails to function correctly.
• Navigating to Fleet Management > Identity & Access > VCF Instances > "VCF-INSTANCE" > Identity Source displays the error:

  An error occurred while trying to get the Directories. Please try again later.

• Attempting to use VCF SSO to log into a vCenter returns a "no healthy upstream" error.
• Checking the vidb-external namespace reveals that vidb-service pods are stuck in an Init:CrashLoopBackoff state:

  vidb-service-<rs>-<hash>           0/1     Init:CrashLoopBackOff

• Inspecting the vidb-installer init container logs (kubectl logs -n vidb-external <pod-name> -c vidb-installer -f) shows the following critical initialization constraint violation:

  Starting to upload service configuration files
  ./vidb_initialization_helper.sh: line 184: /opt/vmware/idm/installation/accesscontrol/config/configkeystore.bcfks: No such file or directory
  ERROR: duplicate key value violates unique constraint "pk_vidb_init_service_configuration"
  DETAIL: Key (key)=(accesscontrol_configkeystore.bcfks) already exists.

• Querying the vcdb database reveals a blank/empty value for the keystore entry:

  psql -U postgres -d vcdb -c "select * from vidb_init_service_configuration where key='accesscontrol_configkeystore.bcfks'"
  
  key                                | value
  -----------------------------------+-------
  accesscontrol_configkeystore.bcfks |
  (1 row)

Note: Resetting the vmware-system-user accounts for the IDB appliances and remediating passwords does not resolve this issue

Environment

• VMware Cloud Foundation (VCF) 9.x
• VMware Identity Broker (VIDB) 9.x

Cause

This issue occurs when the configkeystore.bcfks file is missing or was not persisted during a datastore-full event, while its corresponding configuration entries remain in the database. The idm.groovy initialization script skips the keystore regeneration process if it detects that accesscontrol_application.properties already exists in the vidb_init_service_configuration table. This leads to a failure when the script attempts to reference the non-existent keystore file.

Resolution

Contact Broadcom Support to resolve this issue, as it requires complex database modifications.