Storage providers section is blank in vCenter UI and unable to change storage policies on Virtual Machines

Products

• VMware vSAN
• VMware vCenter Server

Issue/Introduction

• Attempting to change the default storage policy for a vSAN datastore fails with: "Namespace not found for datastore"
• When navigating to vCenter -> Configure -> Storage providers, the section is empty. Following error message is triggered: "The storage service not initialized. Try again later"
• Virtual Machine storage policy may show as "Unknown".
• Changing or applying a storage policy to a Virtual Machine may fail.
  
  
• After restarting the vmware-sps service on the VCSA, the service health shows green. However, storage provider initialization fails.
  
/var/log/vmware/vmware-sps/sps.log :
  YYYY-MM-DDTH:MM:SSZ [Thread-9] ERROR opId=sps-Main-308140-889 com.vmware.vim.sms.StorageManagerImpl - ProviderLoader intialization failed!
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7b63e86 rejected from
java.util.concurrent.ThreadPoolExecutor@3d0dd4d1[Running, pool size = 500, active threads = 500, queued tasks = 2, completed tasks > = 1041]
  at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at com.vmware.vim.sms.task.TaskManagerImpl.scheduleJob(TaskManagerImpl.java:117)
at com.vmware.vim.sms.task.BatchTaskManagerImpl.processQueuedTasks(BatchTaskManagerImpl.java:73)
at com.vmware.vim.sms.StorageManagerImpl.loadPostProcess(StorageManagerImpl.java:384)
at com.vmware.vim.sms.StorageManagerImpl.access$600(StorageManagerImpl.java:130)
at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.loadVasaProviders(StorageManagerImpl.java:351)
at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:292)

Cause

During the initialization of storage providers, incoming jobs are queued while the providers are still loading. Once all providers have completed loading, the SPS service transitions from Yellow to Green status. However, just before updating the service health to the Initialized state, a post-processing step is triggered to execute all queued jobs. By default, the system allows a maximum of 500 jobs in the processing pool and 2 additional jobs in the queue. If the number of jobs exceeds this limit, vCenter encounters this issue.

Resolution

To resolve this issue, increase the sms.threadpool.queueSize:

1. Edit /usr/lib/vmware-vpx/sps/conf/sms.properties
2. Change the parameter

sms.threadpool.queueSize=2
to
sms.threadpool.queueSize=1000

3. Restart sps service:

service-control --stop vmware-sps
service-control --start vmware-sps

This modification raises the threshold for the number of jobs allowed to wait in the processing queue.

Additional Information

Additional symptoms observed:

• In the vCenter UI Virtual Machine's edit settings, hard disks show no info.
• Virtual Machine deployment fails with error "Failed to get vpcapablityobjectmetaadmininfo entries"