High CPU utilization on vCenter Server 8.0
Article ID: 367603
Updated On:
Products
Issue/Introduction
vCenter Server 8.0u1 or later with tiny setup may encounter high CPU utilization some time after deployment.
The CPU usage is high with top command identifying the idmserv+ as a high consumer.
top - HH:MM:SS up DDD days, HH:MM, 1 user, load average: 9.35, 5.38, 3.52
Threads: 4974 total, 7 running, 4964 sleeping, 0 stopped, 3 zombie
%Cpu0 : 89.6/10.4 100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu1 : 79.0/21.0 100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
GiB Mem : 81.5/13.7 [ ]
GiB Swap: 66.8/25.0 [ ]
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
3237 idmserv+ 20 0 13.5g 603.5m 0.0 4.3 0:00.01 S +
3318 idmserv+ 20 0 13.5g 603.5m 49.9 4.3 110994:57 R +
3236 idmserv+ 20 0 3666.6m 712.9m 0.0 5.1 0:00.01 S +
3307 idmserv+ 20 0 3666.6m 712.9m 49.5 5.1 9969:05 R +
You can see Out Of Memory Error in the /var/log/vmware/vc-ws1a-broker/token-service.log similar to:
YYYY-MM-DDTHH:MM:SS ERROR XXXXX.XXXXX.XXXXX:token (vert.x-eventloop-thread-1) [-;-;127.0.0.1;d6ef8ccf-c7cd-4798-83b5-8dbfc492b7e8;-] io.vertx.ext.web.RoutingContext - Unhandled exception in router java.lang.OutOfMemoryError: Java heap space
Or /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log:
YYYY-MM-DDTHH:MM:SS ERROR YYYYY.YYYYY.YYYYY:accesscontrol (ForkJoinPool-10-worker-7) [HWS;-;127.0.0.1;cda649a0-1002-4816-a3ee-ecc08f5b84b8;-;-;-] io.vertx.ext.web.RoutingContext - Unhandled exception in router java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast to class com.vmware.vidm.common.exception.MultiFormattableRuntimeException (java.lang.OutOfMemoryError is in module java.base of loader 'bootstrap'; com.vmware.vidm.common.exception.MultiFormattableRuntimeException is in unnamed module of loader 'app')
Cause
The scheduled token revocation process causing OOM / high CPU utilization.
Resolution
Add the following lines to /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/XX/fs/opt/vmware/idm/initc/services/token/config/application.properties
Note: XX should be the highest number. Please refer to the resolution in "Login to vCenter Server with External Identity Providers for SSO account stops working".
token.delete.expired.tokens.limit=100
revocation.delete.old.tombstones.limit=100
revoke.by.oauthclientid.batch.size=500
and then restart the identity service using:
vmon-cli -r vc-ws1a-broker
This setting lower values will lower the memory footprint during token deletion.
Related KB: Login to vCenter Server with External Identity Providers for SSO account stops working
Feedback
