VCF Identity Broker Service Fails to start due to Database Storage Exhaustion

Products

• VCF Operations

Issue/Introduction

• The VCF Identity Broker (VIDB) service fails to start, resulting in an "Identity Broker encountered an issue during authentication" error.
• Consequently, vCenter and other VCF components are unable to authenticate users via Single Sign-On (SSO), preventing access to the management environment.

Environment

VCF Operations 9.x

Cause

The primary database pod (vidb-postgres-instance-0) experienced disk space exhaustion on the leader partition (e.g., /dev/sdm or /home/postgres/pgdata). This was triggered by excessive OAuth/Refresh token generation, causing the partition to reach 100% capacity and halting service operations.

Resolution

Prerequisite: Ensure a snapshot of the environment is taken before performing any database modifications.

To resolve this issue, perform the following steps to reclaim space and prevent recurrence:

1. Truncate the Token Table: Use the following commands to identify the token load and clear the TOKEN_Token table:
   
   To check token counts:
   

   kubectl exec -n vidb-external vidb-postgres-instance-0 -c postgres -- psql -U postgres -d vcdb -c "SELECT "tokenType", count() AS total, count() FILTER (WHERE expires >= LOCALTIMESTAMP) AS still_valid FROM "TOKEN_Token" GROUP BY 1 ORDER BY 2 DESC;"

   NOTE : If you get  api-tokens count greater than zero in above output, we need to take customer consent. Because If we delete api-tokens customer need to manually generate them.

   To truncate the table (reclaim space): 

   kubectl exec -n vidb-external vidb-postgres-instance-0 -c postgres -- psql -U postgres -d vcdb -c "SET lock_timeout = 5000; TRUNCATE TABLE "TOKEN_Token";"

2. Increase Storage Quota: 
   
   Navigate to Fleet Management > Lifecycle > VCF Management > Components.
   Select Identity Broker and click STORAGE RESIZE.
   Increase the database storage quota (e.g., from 18GB to 22GB).
   
   
3. Optimize Orchestrator Collection Interval: Adjust the Orchestrator adapter collection interval from the default 5 minutes to 1 hour to reduce the rate of token accumulation.

 

A permanent fix has been planned for a future release and the KB will be updated with the version once the release is available.