ThreadPoolTaskScheduler thread gets stuck and no more scheduled jobs are run in the Domain Manager service

Products

• VMware Cloud Foundation

Issue/Introduction

Scheduled tasks in the Domain Manager service don't get executed due to the ThreadPoolTaskScheduler's sole thread becoming unresponsive, impacting various background operations.

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x 

Cause

The Domain Manager service inherits its ThreadPoolTaskScheduler bean from TelemetryAppConfig, which configures a thread pool size of one. Consequently, if this single thread becomes unresponsive or "stuck" while executing any scheduled job (an unexpected but observed scenario), it blocks all subsequent scheduled tasks from running.

Resolution

The user can increase the number of threads in vac-scheduler pool to 5 by updating the below property in the Domain Manager application.properties and restarting the Domain Manager service.

1. SSH into the SDDC Manager appliance with vcf and then elevate to root with su. 
   
   
2. Edit the application.properties file
   
   vi /etc/vmware/vcf/domainmanager/application.properties
   
   
3. Add the following line at the end of the file vac.scheduler.corepoolsize=5

4. Restart the domain manager service systemctl restart domainmanager
   
   systemctl restart domainmanager

Additional Information

PR# 3577632