Storage Policy Service (SPS) shows unhealthy after adding a large number of ESXi hosts with active IOFilters

Products

• VMware vCenter Server

Issue/Introduction

The Storage Policy Service (SPS) status may show as unhealthy in vCenter Server when a large number of ESXi hosts with active IOFilters (VASA Providers) are added to the vCenter inventory in quick succession.
The SPS generates core dumps with the OutOfMemoryError exception, such as:

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "Catalina-utility-3"
SEVERE: Error sending periodic event
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
    at org.apache.catalina.core.StandardServer.startPeriodicLifecycleEvent(StandardServer.java:889)
    ...
Caused by: java.lang.OutOfMemoryError: Java heap space

Environment

VMware Cloud Foundation 9.0

Cause

This issue occurs when:

• A large number of ESXi hosts with active IOFilters (VASA Providers) are added to the vCenter inventory in a short time frame, and
• The vCenter Server instance already manages a large number of certificates in the Storage Monitoring Service (SMS) or TRUSTED_ROOT trust stores.

Under these conditions, the SPS service consumes excessive Java heap memory while processing certificates and IOFilter-related operations, eventually leading to a Java heap space OutOfMemoryError, causing the SPS service to become unhealthy.

Resolution

To recover from this condition, restart the Storage Policy Service (SPS) on the vCenter Server.

Steps to restart SPS : 

You can restart the service using either of the following methods:

Using service-control:

service-control --stop vmware-sps
service-control --start vmware-sps

 

Or using vMon CLI:

vmon-cli -r sps

After restarting the SPS service, verify that the SPS status is healthy in the vCenter Server UI.