VCF SSO is not working on any VCF component

Products

• VCF Operations

Issue/Introduction

When logging into any component like vCenter,VCFO using VCF SSO, the login fails and displays a no healthy upstream error.

This occurs in environments using an Embedded vIDB, In the Identity Source page we see Error:

An error occurred while trying to get the Directories. Please try again later.

Environment

VCF 9.1

Cause

The embedded vIDB broker experiences an OutOfMemory (OOM) condition and is not automatically flagged by the vCenter health system for a restart.

When reviewing the /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log on vCenter server, you will observe the accesscontrol Vert.x event-loop thread blocking, followed by OOM errors:

The accesscontrol Vert.x event-loop thread starts blocking (2.8s→4.9s blocked, threshold is 2s)
Exception published of type OutOfMemoryError
Failed to create a new channel from an accepted socket. java.lang.OutOfMemoryError: Java heap space
Failed during issuing token ... OutOfMemoryError: Java heap space

Resolution

To resolve this issue, manually restart the vIDB broker service on the vCenter Server.

1. Check the vCenter health status by navigating to https://<vidb-fqdn>/vidb/health or port 5480 (the service may falsely report as healthy).

2. Log in to the vCenter Server Appliance (VCSA) via SSH as the root user.

3. Verify the status of the vIDB broker service:

   service-control --status vc-ws1a-broker
   

4. Restart the service to clear the OutOfMemory state:

   service-control --restart vc-ws1a-broker