"Health status changed alarm" for content-library status changing from green to red due to Garbage Collection (GC)

Products

• VMware vCenter Server

Issue/Introduction

The vCenter Server triggers a Health status changed alarm indicating the content-library status has changed from green to red. In many cases, the status automatically reverts from red to green shortly after.

Example events from journalctl

Alarm 'Health status changed alarm' on Datacenters triggered by event 'content-library status changed from green to red'
Alarm 'Health status changed alarm' on Datacenters triggered by event 'content-library status changed from red to green'

 

 

 

Environment

vCenter Server

Cause

This issue is caused by a temporary timeout in the content-library service health check.

During the health check interval, the Java Virtual Machine (JVM) for the Content Library service may perform a Garbage Collection (GC) operation (specifically a "CMS Final Remark" or similar stop-the-world phase). If the GC takes an extended amount of time, the service becomes temporarily unresponsive to API health queries from vmon, leading to a timeout and a "red" health status.

• vmon.log
  

  Reports a timeout when retrieving health XML: Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:16666/cls/cmhealthstatus. Exception: timed out

• vmware-content-library-gc.log
  

  [GC (CMS Final Remark) ... 42.7621679 secs] [Times: user=0.14 sys=0.34, real=42.76 secs]

Resolution

If the content-library status returns to "green" automatically within a short period, this is considered no functional impact on the vCenter Server.
Check the vSphere Client to ensure the Content Library service health has returned to green.