vapi-endpoint status changed from green to yellow for a short time
Article ID: 411834
Updated On:
Products
Issue/Introduction
vapi-endpoint status changed from green to yellow, but quickly recovered back to green.
"VMware vAPI Endpoint Service Health Alarm" might be also triggered.
Environment
vCenter Server 7.X
vCenter Server 8.X
Cause
vapi-endpoint failed to query to cis-license service and turned its health status to YELLOW. If a query failure is temporary, its health status would recover to GREEN again.
# /var/log/vmware/vapi/endpoint/endpoint.log
YYYY-MM-DDThh:mm:ss.###Z | INFO | state-manager1 | CollectedHealthStatusProviderImpl | Computed health status is GREEN.
:::
YYYY-MM-DDThh:mm:ss.###Z | WARN | state-manager1 | ApiInterfacesFactory | Retrieving interfaces for service 60bd3fbb-ef58-4953-8aeb-524ebf16a336\com.vmware.cis.cs.license.slms has failed.
com.vmware.vapi.internal.core.abort.RequestAbortedException: Http request aborted.
at com.vmware.vapi.internal.protocol.common.Util$1.onAbort(Util.java:105) ~[vapi-runtime.jar:?]
at com.vmware.vapi.internal.core.abort.AbortHandleImpl.abort(AbortHandleImpl.java:45) ~[vapi-runtime.jar:?]
at com.vmware.vapi.endpoint.api.TimedApiProvider.lambda$invoke$0(TimedApiProvider.java:58) ~[vapi-endpoint-1.0.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_351]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_351]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_351]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_351]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_351]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_351]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_351]
YYYY-MM-DDThh:mm:ss.###Z | INFO | state-manager1 | StatusInfoFactory | HEALTH YELLOW Failed to refresh 60bd3fbb-ef58-4953-8aeb-524ebf16a336\com.vmware.cis.cs.license.slms vAPI provider.
:::
YYYY-MM-DDThh:mm:ss.###Z | INFO | state-manager1 | CollectedHealthStatusProviderImpl | Computed health status is YELLOW.
:::
YYYY-MM-DDThh:mm:ss.###Z | INFO | state-manager1 | CollectedHealthStatusProviderImpl | Computed health status is GREEN.During Full GC, cis-license service would be unresponsive and query from vapi-endpoint service would also fail.
We could check Full GC time from /var/log/vmware/cis-license/vmware-cis-license-gc.log.#.
YYYY-MM-DDThh:mm:ss.###+0000: 60022340.902: [Full GC (Ergonomics) YYYY-MM-DDThh:mm:ss.###+0000: 60022341.531: [SoftReference, 235 refs, 0.0001126 secs]2025-09-24T10:15:18.693+0000: 60022341.531: [WeakReference, 936 refs, 0.0001093 secs]YYYY-MM-DDThh:mm:ss.###+0000: 60022341.532: [FinalReference, 148 refs, 0.0001360 secs]YYYY-MM-DDThh:mm:ss.###+0000: 60022341.532: [PhantomReference, 0 refs, 26 refs, 0.0000277 secs]YYYY-MM-DDThh:mm:ss.###+0000: 60022341.532: [JNI Weak Reference, 0.0000367 secs][PSYoungGen: 864K->0K(37888K)] [ParOldGen: 125023K->35922K(127488K)] 125887K->35922K(165376K), [Metaspace: 61339K->61162K(135168K)], 11.7888203 secs] [Times: user=0.34 sys=0.41, real=11.79 secs] Resolution
Full GC of cis-license service isn't avoidable, but there are two ways to mitigate this issue.
# Option 1
If current cis-license service is running for a long time and Full GC happens frequently, restarting cis-license service will reset heap allocation and reduce Full GC frequency.
service-control --restart cis-license
# Option 2
If Full GC happens frequently even after restarting cis-license service, increasing heap allocation for cis-license service could be effective.
Feedback
