Storage providers do not populate in vCenter, cannot change storage policies on VM's

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• When attempting to change the default storage policy for a VSAN datastore, you see a message similar to:

 "Namespace not found for datastore"

• Changing or applying a storage policy to a VM/vdisk fails.
• Storage providers do not populate in vCenter. You see an error message similar to:
  "storage service not initialized" 
• After restarting the vmware-sps service on the VCSA, the service health shows green, but in /var/log/vmware/sps/sps.log you see a message similar to:
   

2018-05-14T15:23:37.978Z [Thread-9] ERROR opId=sps-Main-308140-889 com.vmware.vim.sms.StorageManagerImpl - ProviderLoader
intialization failed!
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7b63e86 rejected from
java.util.concurrent.ThreadPoolExecutor@3d0dd4d1[Running, pool size = 500, active threads = 500, queued tasks = 2, completed tasks > = 1041]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at com.vmware.vim.sms.task.TaskManagerImpl.scheduleJob(TaskManagerImpl.java:117)
at com.vmware.vim.sms.task.BatchTaskManagerImpl.processQueuedTasks(BatchTaskManagerImpl.java:73)
at com.vmware.vim.sms.StorageManagerImpl.loadPostProcess(StorageManagerImpl.java:384)
at com.vmware.vim.sms.StorageManagerImpl.access$600(StorageManagerImpl.java:130)
at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.loadVasaProviders(StorageManagerImpl.java:351)
at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:292)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment

Environment

VMware vSAN 6.x

Cause

During the initialization of storage providers, we queue up jobs that were held during the provider loading phase. After all the storage providers have finished loading, and just before we change the health status of the service to initialized state, we do some post processing where we execute all the queued jobs. By default, we have an upper limit of 500 jobs we can process. If more than that need processing we hit this issue.

Resolution

To resolve this issue:

1. Edit /usr/lib/vmware-vpx/sps/conf/sms.properties
2. Change the paramter

sms.threadpool.queueSize=2
to
sms.threadpool.queueSize=1000

3. Restart sps service:

service-control --stop vmware-sps
service-control --start vmware-sps