Not able to view any Storage Providers in vCenter

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Synchronize Storage Providers under vCenter > Configure > Storage Providers fail with error : SPS service is still initializing.
• SPS service is up and running.
• Deploying a new VSAN cluster we see errors like : A general system error occurred: PBM error occurred during PreCreateCheckCallback: Fault cause: pbm.fault.PBMFault ; No VASA Provider for schema namespace (VSAN) found. The storage service is not initialized. Try again later.
• SPS.log : 

YYYY-MM-DDTHH:MM:SS [pool-3-thread-16] ERROR opId=12c1420e-01-01-01 com.vmware.sps.pbm.impl.ProfileProviderImpl - Error during fetchContentAsString
(pbm.fault.PBMFault) {
faultCause = null,
faultMessage = (vmodl.LocalizableMessage) [
(vmodl.LocalizableMessage) {
dynamicType = null,
dynamicProperty = null,
key = com.vmware.pbm.pbmFault.locale,
arg = (vmodl.KeyAnyValue) [
(vmodl.KeyAnyValue) {
dynamicType = null,
dynamicProperty = null,
key = summary,
value = No VASA Provider for schema namespace (VSAN) found.
 

• vsanvp.log :

YYYY-MM-DDTHH:MM:SS INFO vsanvp[19070] [ServiceInstance::RegisterVsanVP opID=W3] Register VSAN VP to SMS service.
YYYY-MM-DDTHH:MM:SS ERROR vsanvp[19070] [ServiceInstance::RegisterVsanVP opID=W3] Failed to register VSAN VP to sms service
Traceback (most recent call last):
File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/ServiceInstance.py", line 43, in RegisterVsanVP
File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/ServiceInstance.py", line 97, in _FindVsanVp
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 583, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 373, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
raise obj # pylint: disable-msg=E0702
PyCppVmomi.sms.fault.QueryExecutionFault: (sms.fault.QueryExecutionFault) {
msg = 'Upgrade is ongoing.',
faultMessage = (vmodl.LocalizableMessage) [
(vmodl.LocalizableMessage) {
key = 'com.vmware.pbm.pbmFault.locale',
arg = (vmodl.KeyAnyValue) [
(vmodl.KeyAnyValue) {
key = 'summary',
value = 'Upgrade is ongoing.'

Environment

VMware vCenter Server 6.x
VMware vCenter Server 7.0.x

Cause

• This issue can happen if the vCenter had a failed upgrade(probably during import stage) but we continued to use that vCenter instead of reverting to a snapshot or backup and retry upgrade.
• That can be verified from /var/log/vmware/upgrade logs.

Resolution

Take a backup/snapshot of the vCenter before proceeding with the steps below :
 

• Open a SSH session to the vCenter.
• Go to /usr/lib/vmware-vpx/sps/conf/.
• Open sms-config.xml file using a file editor (vi).
• Look for the following tag :

<!-- To report the current upgrade state. In fresh install, this will always be false.
Note : This is a system controlled parameter and should not be manually edited -->
<upgradeOngoing>true</upgradeOngoing>
 

• Change that value from true to false.
• Restart SPS service.
• Logout and login back to vCenter and you should be able to see Storage providers.

Additional Information

Impact/Risks:

• New VM creation can fail.
• Storage vMotion fails.
• vCLS deployment fails.