Health status of some vCenter services intermittently changes from green to red due to insufficient swap space

Products

• VMware vCenter Server

Issue/Introduction

• The health status of vCenter services such as sts, sps and vsphere-ui intermittently changes from green to red and back to green. You might receive an SNMP trap stating "(service name) status changed from green to red".
• The sts, sps, vsphere-ui may be stopped. To check service running state, run service-control --status --all.
• The vSphere Client is temporarily unavailable.
• There is a java log file generated around the time the service health status changed to red.
  • The paths to Java log files are as follows :
    • sts : /storage/log/vmware/sso/hs_err_stsinstaller_pid(pid number).log
    • sps : /storage/log/vmware/vmware-sps/java_error(pid number).log
    • vsphere-ui : /storage/log/vmware/vsphere-ui/java_error(pid number).log
  • You can check when the java log file is generated by checking the last modified time of the file using ls command. For exmaple :
    # ls -l /storage/log/vmware/sso/hs_err_stsinstaller_pid280848.log
    -rw-r--r-- 1 sts lwis 58998 Oct 15 04:15 /storage/log/vmware/sso/hs_err_stsinstaller_pid280848.log
• Swap usage is high. To confirm this, run free :
  # free
                 total        used        free      shared  buff/cache   available
  Mem:        21497432    13963404      178428      696824     7355600     6445128
  Swap:       26206204    24614768     1591436 <<< almost 94% used

 

Environment

VMware vCenter 8.0 U2a and later

Cause

The Java thread of the services stopped due to insufficient swap space.
There is a known memory leak issue in vc-ws1a-broker, causing swap space exhaustion. For more information, see KB 396939 vCenter services experiencing OutOfMemoryError: Java heap space, causing the swap memory to be 100% utilized and due to which vCenter backup fails with " Error: database or disk is full".

 

Resolution

Restarting vc-ws1a-broker should temporarily resolve the issue.
# service-control --restart vc-ws1a-broker
Note : Restarting vc-ws1a-broker will affect authentication using external IDP such as Azure AD and Okta.

To prevent the issue from recurring, follow Resolution section of the KB 396939.