vCenter Server alarm "vc-ws1a-broker status changed from green to unknown" caused by OutOfMemoryError

Products

• VMware vCenter Server

Issue/Introduction

• The following alarm is reported in the vSphere Client:
  vc-ws1a-broker status changed from green to unknown
• If it is a temporary issue, the alarm will return to green after 5 minutes, but in the case of the problem described in this KB, the alarm will not return to green even after 5 minutes have passed.
• The following message indicating an out-of-memory error is recorded in /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log:

YYYY-MM-DDThh:mm:ss,### ERROR #######.#####.#####:accesscontrol (ForkJoinPool-##-worker-##) [HWS;-;127.0.0.1;########-####-####-####-############;-;-;-] io.vertx.ext.web.RoutingContext - Unhandled exception in router java.lang.ClassCastException: class java.lang.OutOfMemoryError can not be cast to class com.vmware.vidm.common.exception.MultiFormattableRuntimeException (java.lang.OutOfMemoryError is in module java.base of loader 'bootstrap'; com.vmware.vidm.common.exception.MultiFormattableRuntimeException is in unnamed module of loader 'app')

YYYY-MM-DDThh:mm:ss,### ERROR #######.#####.#####:accesscontrol (ForkJoinPool-##-worker-##) [HWS;-;127.0.0.1;########-####-####-####-############;-;token_client_id;client_credentials] com.vmware.vidm.accesscontrol.resource.auth.TokenResource - Failed during issuing token java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
        at com.vmware.vidm.common.async.ContextPassingExecutor.lambda$wrap$0(ContextPassingExecutor.java:48)
        at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:49)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space

Environment

vCenter Server 8.0 

Cause

Out of memory occurs due to the value of an internal parameter related to the token deletion/revocation schedule.
As a result of the out of memory occurrence, the vc-ws1a-broker service's status is recognized as unknown.

Resolution

VMware engineering are aware of this issue and are currently working on a solution.

As a workaround, follow the steps in the "Resolution" section of KB 378474.

Additional Information

Japanese KB: OutOfMemoryError が原因で vCenter Server に「vc-ws1a-broker のステータスが緑から不明に変更されました」というアラームが表示される