Error: "Internal error occurred" while upgrading Identity Broker from 9.0.2 to 9.1, although the upgrade is successful.

Products

• VCF Operations

Issue/Introduction

After upgrading VMware Cloud Foundation (VCF) Operations and Identity Broker from version 9.0.2 to 9.1, you may observe the following symptoms:

• UI Error: An error message stating "Internal error occurred. Please retry the operation or contact support if the issue persist" appears in the Upgrade task which completes successfully later.
  
• Persistent Status: The Identity Broker status in the Configure VCF SSO page shows Choose deployment mode – In Progress, preventing further configuration.
• Functionality: Despite these UI errors, authentication and login flows continue to work as expected.

Environment

VCF Identity broker 9.1

VCF Operations 9.1

Cause

In multi-VCF instance configurations where an Identity Broker is shared across multiple domains, a code-level issue causes the SSO domain to be created with an incorrect vcf_instance_id during the VCF Operations 9.1 upgrade. This mismatch between the actual VIDB resource ID and the assigned VCF instance ID in the backend database results in the UI inconsistencies.

Resolution

This issue requires a manual update to the VCF Operations database to correct the mapping.

Prerequisites

• Take a snapshot of the VCF Operations nodes before making the following changes.

Step 1: Identify the Mismatched Records

1. SSH to the VCF Operations primary node.
2. Connect to the VCF Operations database:

   su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -p 5433 -d vcopsdb"

3. Retrieve the current SSO domain records:

   SELECT * FROM kv_vidb_sso_domain;

4. Identify the vcf_instance_id, vidb_resource_id, and key for the affected entry.
5. Use the VCF Operations API to find the correct VCF Instance ID for your VIDB:

   GET https://<ops-ip>/suite-api/api/fleet-management/iam/vidbs

6. Compare the vcfInstanceId from the API output with the vcf_instance_id retrieved from the database.

Step 2: Update the Database Mapping

1. For each record where the IDs do not match, run the following update command (replace placeholders with actual values from Step 1):

   UPDATE kv_vidb_sso_domain SET vcf_instance_id = '<correct-vcf-instance-id>', col_kv_strvalue = replace(col_kv_strvalue, '<existing-mismatched-id>', '<correct-vcf-instance-id>') WHERE key = '<target-key>';

2. Verify the change in the database:

   SELECT * FROM kv_vidb_sso_domain WHERE key = '<target-key>';

Step 3: Verify the UI

1. Log in to the VCF Operations UI.
2. Navigate to VCF SSO Overview.
3. Confirm that the status now shows as Completed and the Installed Location correctly reflects the management domain.