vapi-endpoint status changed from green to red and back to green quickly

Products

• VMware vCenter Server

Issue/Introduction

• The alarm "vapi-endpoint status changed from green to red" occurred.
• The vapi-endpoint service status goes red, then quickly returns to green.
  journalctl -b -0
  

  [timestamp] [hostname] vpxd[25927]: Event [728723725] [1-1] [timestamp] [vim.event.HealthStatusChangedEvent] [info] [Vmonuser] [] [728723725] [vapi-endpoint status changed from green to red]
  [timestamp] [hostname] vpxd[25927]: Event [728724618] [1-1] [timestamp] [vim.event.EventEx] [info] [] [] [728724618] [Alarm 'Health status changed alarm' on Datacenters triggered by event 728724616 'vapi-endpoint status changed from red to green']

Environment

vCenter Server 7.X
vCenter Server 8.X

Cause

vmon health check for vapi-endpoint is timed out.

/var/log/vmware/vmon/vmon.log

[timestamp] Wa(03) host-13814 <vapi-endpoint> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:XXXXX/vapiendpoint/health. Exception: timed out

At the same time, you can see that the vapi-endpoint service is running a full GC.

/var/log/vmware/vapi/endpoint/vmware-vapi-endpoint-gc.log.N.current

[timestamp]: 62921406.790: [Full GC (Ergonomics) [timestamp]: 62921416.708: [SoftReference, 1094 refs, 0.0001464 secs][timestamp]: 62921416.708: [WeakReference, 1739 refs, 0.0001429 secs][timestamp]: 62921416.708: [FinalReference, 531 refs, 0.0588955 secs][timestamp]: 62921416.767: [PhantomReference, 0 refs, 241 refs, 0.0000410 secs][timestamp]: 62921416.767: [JNI Weak Reference, 0.0000324 secs][PSYoungGen: 1024K->0K(30720K)] [ParOldGen: 152712K->81260K(133632K)]153736K->81260K(164352K), [Metaspace: 83501K->83021K(184320K)], 49.0266726 secs] [Times: user=0.45 sys=0.39, real=49.03 secs]

Resolution

Full GC cannot be avoided, so it will inevitably occur if the timing matches.
This is not abnormal behavior and can be ignored as the next health check will be successful.