Login to vCenter Server with External Identity Providers for SSO account stops working

Products

• VMware vCenter Server

Issue/Introduction

• You can configure an external identity provider for federated authentication.

  vSphere 8.0 Update 1 and later supports Okta. vSphere 8.0 Update 2 and later supports Microsoft Entra ID (formerly called Azure AD). Starting in vSphere 8.0 Update 3, vSphere supports PingFederate.

• While login to vCenter Server with External SSO account, the following error is thrown:

"Error: Workspace ONE Access encountered an error. Contact your  admin and provide the information displayed below" 

• Restarting vCenter Server services temporary solves the issue. 
• in log file /var/log/vmware/vc-ws1a-broker/token-service.log, you can see line similar to 
  

  YYYY-MM-DDTHH:MM:SS ERROR XXXXX.XXXXX.XXXXX:token (vert.x-eventloop-thread-1) [-;-;127.0.0.1;d6ef8ccf-c7cd-4798-83b5-8dbfc492b7e8;-] io.vertx.ext.web.RoutingContext - Unhandled exception in router java.lang.OutOfMemoryError: Java heap space

  
  Or /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log:

  YYYY-MM-DDTHH:MM:SS ERROR YYYYY.YYYYY.YYYYY:accesscontrol (ForkJoinPool-10-worker-7) [HWS;-;127.0.0.1;cda649a0-1002-4816-a3ee-ecc08f5b84b8;-;-;-] io.vertx.ext.web.RoutingContext - Unhandled exception in router java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast to class com.vmware.vidm.common.exception.MultiFormattableRuntimeException (java.lang.OutOfMemoryError is in module java.base of loader 'bootstrap'; com.vmware.vidm.common.exception.MultiFormattableRuntimeException is in unnamed module of loader 'app')

  
  
  

 

Environment

• vCenter Server 8.0 

Cause

This issue is happening during a token revocation schedule. When the process runs, it pulls entries from the DB with a default page size.
This causes the Out Of Memory.
This issue also occurs when external SSO is not configured.

Resolution

VMware engineering are aware of this issue and are currently working on a solution. In the meantime, to work around the issue please follow the steps listed below".

 

1. Take a snapshot of vcenter

VMware recommends to have an offline Snapshot  (virtual machine powered off) of all nodes in the same SSO domain "in ELM" before any activity that will include changes in the vCenter Server.

https://knowledge.broadcom.com/external/article/313886/vmware-vcenter-in-enhanced-linked-mode-p.html

 

2.  Identify most token/config/application.properties file

#cd /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots

#find -iname application.properties | grep token

it would be similar to 

./11/fs/opt/vmware/idm/initc/services/token/config/application.properties
./12/fs/opt/vmware/idm/initc/services/token/config/application.properties
./33/fs/opt/vmware/idm/initc/services/token/config/application.properties

 

3.  Backup application.properties from the highest snapshot number from the previous command  ./XX/fs/opt/vmware/idm/initc/services/token/config/application.properties  (in the previous example XX would be 33)

#cp ./XX/fs/opt/vmware/idm/initc/services/token/config/application.properties /var/core/application.properties

 

4. Edit ./XX/fs/opt/vmware/idm/initc/services/token/config/application.properties

#vi ./XX/fs/opt/vmware/idm/initc/services/token/config/application.properties

 

5. Add the following lines in bold  under token configuration

it would be similar to:

## Token configuration
token.revocation.check.source=local
token.public.key.source=remote_crypto

token.delete.expired.tokens.limit=100
revocation.delete.old.tombstones.limit=100
revoke.by.oauthclientid.batch.size=500

 

6.  Restart broker service

#service-control --restart vc-ws1a-broker