Failed to log into vCenter with Entra ID as the Identity Provider

Products

VMware vCenter Server

Issue/Introduction

vCenter configured to use Entra ID for authentication, AD users failed to login with the user account. Error message of "Error communicating with external service".
/var/log/vmware/vc-ws1a-broker/token-service.log shows

Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)

YYYY-MM-DDThh:mm:ss,323 WARN <vCenter FQDN>:federation (ForkJoinPool-2-worker-2) [-;-;-;-;-;-] com.vmware.vidm.common.gateway.mesh.GatewayAuthProvider - Failed to acquire token, returning cached token - Optional.empty, io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114
YYYY-MM-DDThh:mm:ss,325 INFO <vCenter FQDN>:federation (main) [-;-;-;-;-;-] com.vmware.vidm.federation.cds.TenantFeatureProvider - Creating tenant feature cache with ttl seconds: 600, max size: 10000

YYYY-MM-DDThh:mm:ss,744 INFO <vCenter FQDN>:federation (vert.x-eventloop-thread-25) [ -;-;-;-;-;- ] org.bouncycastle. jsse. provider. ProvTlsClient - [client ###### @63####fd] disconnected from login.microsoftonline.com:443
YYYY-MM-DDThh:mm:ss,033 INFO <vCenter FQDN>:federation (Gateway-Token-Refresher-#########) [ -;-;-;-;-;- ] com. vmware. vidm. common. gateway.mesh. Gateway AuthProvider - Invalid Token - [Now: YYY-MM-DDThh:mm:ss.#########Z] - GatewayToken [Hash : ######### , Expiry: YYYY-MM-DDThh:mm:ssZ] [Errors : 0]
YYYY-MM-DDThh:mm:ss,033 INFO <vCenter FQDN>:federation (Gateway-Token-Refresher-#########) [ -;-;-;-;-;- ] com. vmware. vidm. common. gateway.mesh. GatewayAuthProvider - Invalid Token - [Now: YYY-MM-DDThh:mm:ss.#########Z] - GatewayToken [Hash : ######### , Expiry: YYYY-MM-DDThh:mm:ssZ] [Errors : 0]
YYYY-MM-DDThh:mm:ss,146 INFO <vCenter FQDN>:federation (ForkJoinPool-2-worker-######) [ -;-;-;-;-;- ] com. vmware. vidm. common. gateway . mesh. GatewayAuthProvider - Successfully acquired - Optional [GatewayToken [Hash : ######### , Expiry: YYYY-MM-DDThh:mm:ssZ] [Errors : 0] ]

YYYY-MM-DDThh:mm:ss,744 INFO <vCenter FQDN>:federation(vert.x-eventloop-thread-25)[ -;-;-;-;-;- ] org.bouncycastle.jsse.provider.ProvTlsClient - [client ###### @63####fd] disconnected from <federation-domain>:443
YYYY-MM-DDThh:mm:ss,033 INFO <vCenter FQDN>:federation(Gateway-Token-Refresher-#########)[ -;-;-;-;-;- ] com.vmware.vidm.common.gateway.mesh.Gateway AuthProvider - Invalid Token - [Now: YYYY-MM-DDThh:mm:ss.#########Z] - GatewayToken [Hash : #########,Expiry: YYYY-MM-DDThh:mm:ssZ] [Errors:0]
YYYY-MM-DDThh:mm:ss,033 INFO <vCenter FQDN>:federation(Gateway-Token-Refresher-#########)[ -;-;-;-;-;- ] com.vmware.vidm.common.gateway.mesh.Gateway AuthProvider - Invalid Token - [Now: YYYY-MM-DDThh:mm:ss.##########] - GatewayToken [Hash:#########,Expiry: YYYY-MM-DDThh:mm:ssZ] [Errors:0]
YYYY-MM-DDThh:mm:ss,146 INFO <vCenter FQDN>:federation (ForkJoinPool-2-worker-######) [ -;-;-;-;-;- ] com.vmware.vidm.common.gateway .mesh.GatewayAuthProvider - Successfully acquired - Optional [GatewayToken [Hash:#########,Expiry:YYYY-MM-DDThh:mm:ssZ] [Errors:0]]

Environment

vCenter 8.x
vCenter 9.0.0

Cause

Internal token refresh expires every 6 hours and it failed to refresh the internal token.

YYYY-MM-DDTXX,110 WARN #######.example.com:federation (vert.x-worker-thread-4) [CUSTOMER;-;127.0.0.1;########-####-####-####-############;-;-] com.vmware.vidm.common.vertx.auth.RequestContextResolutionHandler - Token validation (Signature & Revocation) [Id: ########-####-####-####-############, isValid: false, isExpired: true]
YYYY-MM-DDTXX,111 WARN #######.example.com:federation (vert.x-eventloop-thread-3) [CUSTOMER;-;127.0.0.1;########-####-####-####-############;-;-] com.vmware.vidm.common.vertx.exception.handler.DefaultExceptionHandler - Request failed without exception, status_code: 401
YYYY-MM-DDTXX,198 WARN #######.example.com:federation (vert.x-worker-thread-3) [CUSTOMER;-;127.0.0.1;########-####-####-####-############;-;-] com.vmware.vidm.common.vertx.auth.RequestContextResolutionHandler - Token validation (Signature & Revocation) [Id: 3########-####-####-####-############, isValid: false, isExpired: true]
YYYY-MM-DDTXX,199 WARN #######.example.com:federation (vert.x-eventloop-thread-3) [CUSTOMER;-;127.0.0.1;########-####-####-####-############;-;-] com.vmware.vidm.common.vertx.exception.handler.DefaultExceptionHandler - Request failed without exception, status_code: 401

Resolution

This issue is resolved in below versions

vCenter Server 8.0.U3g - Log in to the Broadcom Support Portal to download this patch.
vCenter 9.0.1 - Log in to the Broadcom Support Portal to download this patch, depending on entitlement, VMware vSphere Foundation or VMware Cloud Foundation

Workaround this issue follow below steps:

1. To resolve the issue, follow these steps to restart the vc-ws1a-broker service in vCenter:
2. SSH into the vCenter Server using root credentials.
3. Run the following command to restart the broker service:
  - vmon-cli --restart vc-ws1a-broker