sts status changed from green to red alarm triggered in vCenter Server

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

• The health status of the STS (Security Token Service) service temporarily changes to "RED" (critical) and returns to Green shortly after
• A fatal error (Out of Memory) indicating a failure to allocate memory for the Java Runtime is logged in sts-health-status.log (/var/log/vmware/sso/sts-health-status.log):
  

  # There is insufficient memory for the Java Runtime Environment to continue.
  # Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate
  ...
  OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(...) failed; error='Cannot allocate memory' (errno=12)
  ...
  Failed to acquire token after 1.034682 seconds.
  Publishing health status as RED to vMon.

• Logs indicating that the STS service exited with Exit code 143 and was subsequently restarted automatically are recorded in vmon.log (/var/log/vmware/vmon/vmon.log):
  

  Wa(03) host-12095 <sts> Service exited. Exit code 143
  In(05) host-12095 <sts-prestart> Constructed command: ...
  In(05) host-12095 <sts> Service STARTED successfully.

• Multiple Java Runtime Environment (JRE) crash report files are found in the SSO log directory (/var/log/vmware/sso/) with the following naming convention:

  hs_err_sts_pid<PID>.log
  hs_err_stsinstaller_pid<PID>.log

   Inside these hs_err files, a fatal error indicating a native memory allocation failure is present:

  # There is insufficient memory for the Java Runtime Environment to continue.
  # Native memory allocation (malloc) failed to allocate xxxxx bytes for Chunk::new
  # Possible reasons:
  #   The system is out of physical RAM or swap space
  #   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap

Environment

VMware vCenter Server

Cause

This issue occurs when the vCenter Server Appliance (VCSA) experiences a temporary Out of Memory (OOM) condition due to a sudden workload spike.

The physical memory exhaustion prevents the STS Java process from allocating required memory, causing it to crash and report a "RED" health status. Additionally, environments experiencing this issue often show high Swap space usage, indicating a chronic memory shortage.

Resolution

The vCenter Server service management (vMon) automatically restarts the STS service to restore it to a normal state (Green).

Workaround:
If this is an isolated incident, the alarm can be safely ignored once the service is running normally.

Resolution (Recommended):
If the VCSA consistently uses high Swap space, it indicates a chronic memory shortage. Increasing the physical memory (RAM) of the VCSA is highly recommended to prevent recurrence.