Storage providers do not populate in vCenter, cannot change storage policies on VM's
search cancel

Storage providers do not populate in vCenter, cannot change storage policies on VM's

book

Article ID: 326589

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:

  • When attempting to change the default storage policy for a VSAN datastore, you see a message similar to:
 "Namespace not found for datastore"
  • Changing or applying a storage policy to a VM/vdisk fails.
  • Storage providers do not populate in vCenter. You see an error message similar to:
    "storage service not initialized" 
  • After restarting the vmware-sps service on the VCSA, the service health shows green, but in /var/log/vmware/sps/sps.log you see a message similar to:
     
2018-05-14T15:23:37.978Z [Thread-9] ERROR opId=sps-Main-308140-889 com.vmware.vim.sms.StorageManagerImpl - ProviderLoader
intialization failed!
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7b63e86 rejected from
java.util.concurrent.ThreadPoolExecutor@3d0dd4d1[Running, pool size = 500, active threads = 500, queued tasks = 2, completed tasks > = 1041]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at com.vmware.vim.sms.task.TaskManagerImpl.scheduleJob(TaskManagerImpl.java:117)
at com.vmware.vim.sms.task.BatchTaskManagerImpl.processQueuedTasks(BatchTaskManagerImpl.java:73)
at com.vmware.vim.sms.StorageManagerImpl.loadPostProcess(StorageManagerImpl.java:384)
at com.vmware.vim.sms.StorageManagerImpl.access$600(StorageManagerImpl.java:130)
at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.loadVasaProviders(StorageManagerImpl.java:351)
at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:292)


Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment



Environment

VMware vSAN 6.x

Cause


During the initialization of storage providers, we queue up jobs that were held during the provider loading phase. After all the storage providers have finished loading, and just before we change the health status of the service to initialized state, we do some post processing where we execute all the queued jobs. By default, we have an upper limit of 500 jobs we can process. If more than that need processing we hit this issue.

Resolution


To resolve this issue:
  1. Edit /usr/lib/vmware-vpx/sps/conf/sms.properties
  2. Change the paramter
sms.threadpool.queueSize=2
to
sms.threadpool.queueSize=1000
  1. Restart sps service:
service-control --stop vmware-sps
service-control --start vmware-sps


Additional Information

Additional symptoms observed:

  • In the vCenter UI VM's edit settings, hard disks were showing no "info/data"
  • Virtual Machine deployment was failing with error "Failed to get vpcapablityobjectmetaadmininfo entries"