Unable to login to vCenter due to Out of Memory condition in vc-ws1a-broker service with Microsoft Entra as Identity provider.
search cancel

Unable to login to vCenter due to Out of Memory condition in vc-ws1a-broker service with Microsoft Entra as Identity provider.

book

Article ID: 425790

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Users may experience authentication failures when attempting to log in to the vCenter Server UI using Active Directory (AD) credentials in environments configured with Microsoft Entra (formerly Azure AD) as the Identity Provider (IDP) 
  • Login attempts fail with the following error message:
    • [400] Unable to authenticate. Check your credentials. If problem persists, contact your administrator.

  • The authentication requests to the external AD server are successful, but the process subsequently times out.
    • From /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log
      • YYYY-MM-DDTHH:MM:SS INFO  <vCenter_FQDN>:federation (federation-business-pool-0) [CUSTOMER;-;<IP_Address>;3a#####90-####-4b9d-####-######a7d9;-;-] com.vmware.vidm.federation.login.context.LoginContextManager - Created new login context with id: e####4da-####-####-a21c-#######61f1
        YYYY-MM-DDTHH:MM:SS INFO  <vCenter_FQDN>:federation (federation-business-pool-0) [CUSTOMER;-;<IP_Address>;3a#####90-####-4b9d-####-######a7d9;-;e####4da-####-####-a21c-#######61f1] com.vmware.vidm.federation.authenticator.oidc.OidcAuthenticationBaseService - Constructing OidcOauthProvider, getting oidc configuration for identity provider Id 0e######6-####-####-####-#########d7c1
        YYYY-MM-DDTHH:MM:SS INFO  <vCenter_FQDN>:federation (federation-business-pool-0) [CUSTOMER;-;<IP_Address>;1e#####4b-####-4f01-####-######39fb;-;-] com.vmware.vidm.federation.login.context.LoginContextManager - Loaded login context: e####4da-####-####-a21c-#######61f1
        YYYY-MM-DDTHH:MM:SS INFO  <vCenter_FQDN>:federation (federation-business-pool-0) [CUSTOMER;-;<IP_Address>;1e#####4b-####-4f01-####-######39fb;-;e####4da-####-####-a21c-#######61f1] com.vmware.vidm.federation.utils.MetricsPublisherUtil - OIDC authentication successful
        YYYY-MM-DDTHH:MM:SS INFO  <vCenter_FQDN>:federation (federation-business-pool-0) [CUSTOMER;-;<IP_Address>;1e#####4b-####-4f01-####-######39fb;-;e####4da-####-####-a21c-#######61f1] com.vmware.vidm.federation.login.processor.AuthResponseUserResolver - Fetching user for jit login context: e####4da-####-####-a21c-#######61f1 on attribute ExternalId=6#####fb-####-####-bdf3-########e409, domains: [DOMAIN.COM]
        YYYY-MM-DDTHH:MM:SS INFO  <vCenter_FQDN>:federation (federation-business-pool-0) [CUSTOMER;-;<IP_Address>;1e#####4b-####-4f01-####-######39fb;-;e####4da-####-####-a21c-#######61f1] com.vmware.vidm.federation.token.SessionTokenGenerator - Generating token with Id: 1####1f7-####-####-98f6-#######6b18 for user 4######70-####-####-####-#######abce with expiry: 1768026002 for contextId: e####4da-####-####-a21c-#######61f1
        YYYY-MM-DDTHH:MM:SS INFO  <vCenter_FQDN>:federation (federation-business-pool-0) [CUSTOMER;-;<IP_Address>;1e#####4b-####-4f01-####-######39fb;-;e####4da-####-####-a21c-#######61f1] com.vmware.vidm.federation.token.TokenPersistenceService - Generated token with Id 1d1921f7-f5ac-4d48-98f6-f2b7ea996b18 and calling rest endpoint to add to Token service
        YYYY-MM-DDTHH:MM:SS INFO  <vCenter_FQDN>:federation (federation-business-pool-0) [CUSTOMER;-;<IP_Address>;1e#####4b-####-4f01-####-######39fb;-;e####4da-####-####-a21c-#######61f1] com.vmware.vidm.federation.login.LoginEventServiceAspect - Login successful for context: e####4da-####-####-a21c-#######61f1 and username: Optional[<USER_NAME>] domain: Optional[Optional[DOMAIN.COM]], resourceUuid: Optional.empty

  • Later, the request is getting timeout. 
    • From /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log
      • YYYY-MM-DDTHH:MM:SS ERROR qlctcvd00229.merck.com:accesscontrol (ForkJoinPool-2-worker-48810) [-;-;-;-;-;-;-] com.vmware.vidm.accesscontrol.AcsServiceAuth
        Provider - Failed to acquire ACS Service token. com.vmware.vidm.accesscontrol.exceptions.oauth2.TokenGenerationException: oauth2.token.encode.failed
                at com.vmware.vidm.accesscontrol.exceptions.oauth2.TokenGenerationException.failedToEncodeTokenForSigning(TokenGenerationException.java:45)
                at com.vmware.vidm.accesscontrol.TokenSigningService.lambda$signJwtWithOtaAuth$5(TokenSigningService.java:97)
                at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture.postFire(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
                at com.vmware.vidm.common.async.ContextPassingExecutor.lambda$wrap$0(ContextPassingExecutor.java:48)
                at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
        Caused by: io.vertx.core.http.impl.NoStackTraceTimeoutException: The timeout of 20000 ms has been exceeded when getting a connection to localhost:10111
  • Also, 
    • From /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log
      • YYYY-MM-DDTHH:MM:SS ERROR <vCenter_FQDN>:accesscontrol (ForkJoinPool-2-worker-48731) [CUSTOMER;-;127.0.0.1;0#####c-842e-####-####-########321a;-;tgIc#############################msi;password] com.vmware.vidm.accesscontrol.tokengranter.password.FederationPasswordTokenGranter - Error while validating user password for user <[email protected]>. java.lang.OutOfMemoryError: Java heap space
        YYYY-MM-DDTHH:MM:SS WARN  <vCenter_FQDN>:accesscontrol (ForkJoinPool-2-worker-48731) [CUSTOMER;-;127.0.0.1;0#####c-842e-####-####-########321a;-;tgIc#############################msi;password] com.vmware.vidm.accesscontrol.resource.auth.TokenResource - Failed during issuing token java.util.concurrent.CompletionException: com.vmware.vidm.accesscontrol.exceptions.ServiceCommunicationException: service.communication.error
                at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture.postFire(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
                at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
                at com.vmware.vidm.common.async.ContextPassingExecutor.lambda$wrap$0(ContextPassingExecutor.java:48)
                at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
                at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
        Caused by: com.vmware.vidm.accesscontrol.exceptions.ServiceCommunicationException: service.communication.error
                at com.vmware.vidm.accesscontrol.tokengranter.password.FederationPasswordTokenGranter.lambda$processPasswordGrant$6(FederationPasswordTokenGranter.java:168)
                ... 13 more
        Caused by: java.lang.OutOfMemoryError: Java heap space

Environment

  • VMware vCenter Server 7.0
  • VMware vCenter Server 8.0

Cause

The vc-ws1a-broker service, which handles the federation and token exchange between vCenter and the Identity Provider, has exhausted its allocated Java Heap Space. When the service encounters an OutOfMemoryError (OOM), it becomes unable to process new token encoding requests, leading to connection timeouts.

Resolution

To resolve this issue, you must restart the vc-ws1a-broker service to clear the exhausted memory and re-initialize the Java heap.

Steps to restart vc-ws1a-broker server:

  • Log in to the vCenter Server Appliance (VCSA) via SSH as root.
  • Run the following command to restart the service: 
    •  service-control --stop vc-ws1a-broker && service-control --start vc-ws1a-broker
  • Verify the service status: 
    • service-control --status vc-ws1a-broker
  • Attempt to log in to the vCenter UI again.

Additional Information

If this issue recurs frequently, it may be necessary to increase the maximum heap memory for the vc-ws1a-broker service. Please contact Broadcom/VMware Support for specific guidance on modifying service configuration files.

Powered by Wolken Software