VAMI reports Appliance has increased swap activity health warning

Products

• VMware vCenter Server

Issue/Introduction

When accessing the Virtual Appliance Management Interface (VAMI), the system reports a health warning stating: "Appliance has increased swap activity. Add more memory to the machine."

Symptoms include:

• There is no functional impact to the vCenter Server.

• The alert may self-heal but returns intermittently, often on systems with high uptime.

• The top command indicates the idmserver process is consuming high memory:

  • 

• Virtual Memory Size (VSZ), the process reserves very high virtual address space. While this number is alarmingly high, the actual physical memory (RSS) used is very low. (VSZ and RSS can be retrieved from var/log/vmware/procstate) 

  • For example:

  • | PID      | User         | CPU    | VSZ        | RSS        | Service

    -------------------------------------------------------------------------------------

    | 16183    | idmserv+     | 47.2%  | 17.44 GB   | 0.45 GB    | accesscontrol (IDM)

• The massive VSZ is likely an artifact of modern Java 17+ memory allocation on Linux. 

• Bash script analysis of /proc/smaps confirms high SWAP usage attributed to the java process associated with idmserver.

• printf "%-10s %-15s %s\n" "SWAP(KB)" "PID" "SERVICE NAME"; printf "%-40s\n" "--------------------------------------------------------"; for pid in $(ls /proc | grep -E '^[0-9]+$'); do if [ -f /proc/$pid/smaps ]; then swap=$(awk '/^Swap:/ {sum+=$2} END {print sum+0}' /proc/$pid/smaps); if [ "$swap" -gt 0 ] 2>/dev/null; then name=$(ps -p $pid -o comm=); printf "%-10s %-15s %s\n" "$swap" "$pid" "$name"; fi; fi; done | sort -n -r | head -20

  • For example:

  • SWAP      PID     SERVICE NAME
11741504 16117    java 

• Errors observed in /var/log/vmware/vc-ws1a-broker/token-service.log:

YYYY-MM-DDTTHH:MM:SS,544 WARN  vcsa-############token (ForkJoinPool-2-worker-2) [;;;;] com.vmware.vidm.common.resiliency.circuitbrea
ker.CircuitBreakers - Exception during execution inside circuit breaker LOCALHOST java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$
AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114
        at com.vmware.vidm.common.http.client.vertx.VertxHttpClient.handleException(VertxHttpClient.java:224)
        at com.vmware.vidm.common.http.client.vertx.VertxHttpClient.lambda$execute$0(VertxHttpClient.java:82)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
        at com.vmware.vidm.common.async.ContextPassingExecutor.lambda$wrap$0(ContextPassingExecutor.java:48)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)

YYYY-MM-DDTTHH:MM:SS,545 WARN  vcsa-############:token (ForkJoinPool-2-worker-2) [;;;;] com.vmware.vidm.common.gateway.mesh.GatewayAu
thProvider - Failed to acquire token, returning cached token - Optional.empty, io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114

• Errors observed in /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log

com.vmware.vidm.common.gateway.mesh.GatewayAuthProvider - Invalid Token - [Now: YYYY-12-24T16:56:50.031981249Z] - GatewayToken[Hash:1804329599 , Expiry:YYYY-12-24T01:17:08Z][Errors:0]

 

 

Environment

vCenter Server 8.0 (Prior to 8.0 U3g).

 

Cause

This issue is due to internal expired tokens. Since, idmserver is attempting to get new tokens every minute, which causes an additional load. While tokens are normally refreshed every six hours, this frequent attempt eventually exhausts the threads and causes the alert. 

Resolution

This issue is resolved in vCenter Server 8.0 U3g.

To work around this issue on versions prior to 8.0 U3g, restart the vc-ws1a-broker service:

1. Connect to the vCenter Server Appliance via SSH.

2. Run the following command to restart the service: vmon-cli --restart vc-ws1a-broker

Note:

• Restarting this service will not cause any operational impact.
• It will take about an hour for the alert to subside. 

Additional Information

vSphere vCenter Server 8.0 U3g Release Notes