ERROR: no healthy upstream while login to vCenter using domain accounts

Products

• VMware vCenter Server 8.0

Issue/Introduction

Unable to login using domain accounts while vCenter is configured to use Entra ID for the Identity Provider ERROR: no healthy upstream

Logs snipped:
/var/log/vmware/vc-ws1a-broker/federation-service.log

[YYYY-MM-DDTHH:MM:SS] WARN  FQDN.Domain.com:federation (ForkJoinPool-2-worker-40845) [-;-;-;-;-;-] com.vmware.vidm.common.resiliency.circuitbreaker.CircuitBreakers - Exception during execution inside circuit breaker LOCALHOST java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114
        at com.vmware.vidm.common.http.client.vertx.VertxHttpClient.handleException(VertxHttpClient.java:224)
        at com.vmware.vidm.common.http.client.vertx.VertxHttpClient.lambda$execute$0(VertxHttpClient.java:82)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
        at com.vmware.vidm.common.async.ContextPassingExecutor.lambda$wrap$0(ContextPassingExecutor.java:48)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)

[YYYY-MM-DDTHH:MM:SS] WARN  FQDN.Domain.com:federation:federation (ForkJoinPool-2-worker-40845) [-;-;-;-;-;-] com.vmware.vidm.common.gateway.mesh.GatewayAuthProvider - Failed to acquire token, returning cached token - Optional[GatewayToken[Hash:704323238] , Expiry:[YYYY-MM-DDTHH:MM:SS][Errors:0]], io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114

Cause

The issue appears to be caused by a known problem with the vCenter token refresh mechanism. When retrieving the global configuration (which includes the "restTokenPublishingEnabled" property) via an HTTP call, a race condition can occur if the call times out. This issue is exacerbated by a 1-hour TTL (Time-To-Live) for the cache, causing repeated refresh attempts and potential blocking. 

Resolution

This is a know issue with vSphere 8.0.
The issue is resolved in vCenter 8.0 Update 3g.

Workaround:
Restart of vCenter services.

"service-control --stop --all && service-control --start --all"