Aria Operations Scheduled Reports Fail to Generate or Send

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Scheduled reports in VMware Aria Operations do not generate or send.
• Manual report generation works correctly.
• The issue is not specific to any particular report template.
• In the /storage/log/vcops/log/callstack/analytics-.log, you will see DistTaskReportRenderDistributedScheduler thread in a waiting state

"DistTaskReportRenderDistributedScheduler" Id=######## in WAITING on lock=java.util.concurrent.ForkJoinTask$AdaptedRunnableAction@44d04530

• In the gemfire analytics log (/storage/log/vcops/log/'gemfire_vRealize Ops Analytics_####.log') you will see struck threads

WARN [ThreadsMonitor tid=31] org.apache.geode.internal.monitoring.ThreadsMonitoringProcess.lambda$mapValidation$1 - Thread ##### (0x0) "ServerConnection on port 10000 Thread #####" may be stuck

Environment

Aria operations 8.18.5 and below

Cause

A Known JDK ForkJoinPool defect hangs the shared pool used by KvStoreOldToNewWrapper, the report scheduler's user-lookup call blocks on that same pool preventing any scheduled reports from queuing cluster-wide until a full cluster is restarted

Resolution

This issue is fixed in VMware Aria Operations 8.18.6 and higher.

Workaround

If an immediate upgrade is not possible, perform a cluster offline-online operation to clear the stuck thread pool:

1. Log in to the Aria Operations Admin UI (https://<FQDN>/admin).
2. Click Take Cluster Offline.
3. Once the status shows Offline, click Bring Cluster Online.