Not able to view any Storage Providers in vCenter

Products

• VMware vCenter Server

Issue/Introduction

• Synchronize Storage Providers under vCenter > Configure > Storage Providers fail with error : SPS service is still initializing.
  
  
• Adding or modifying hard disks of virtual machine fails with errors like: A general system error occurred: PBM error occurred during PreReconfigureCallback: Fault cause: pbm.fault.PBMFault;Error when trying to run pre-provision validation;No VASA Provider for schema namespace (VSAN) found.
  
  
• SPS service is up and running.
  
  
• Deploying a new VSAN cluster we see errors like : A general system error occurred: PBM error occurred during PreCreateCheckCallback: Fault cause: pbm.fault.PBMFault ; No VASA Provider for schema namespace (VSAN) found. The storage service is not initialized. Try again later.


• From /var/log/vmware/sps.log in vCenter: 

YYYY-MM-DDTHH:MM:SS [pool-3-thread-16] ERROR opId=12c1420e-01-01-01 com.vmware.sps.pbm.impl.ProfileProviderImpl - Error during fetchContentAsString
(pbm.fault.PBMFault) {
faultCause = null,
faultMessage = (vmodl.LocalizableMessage) [
(vmodl.LocalizableMessage) {
dynamicType = null,
dynamicProperty = null,
key = com.vmware.pbm.pbmFault.locale,
arg = (vmodl.KeyAnyValue) [
(vmodl.KeyAnyValue) {
dynamicType = null,
dynamicProperty = null,
key = summary,
value = No VASA Provider for schema namespace (VSAN) found.

• Upon checking the /usr/lib/vmware-vpx/sps/conf/sms-config.xml file we see the if the parameter is true:  
  
  <upgradeOngoing>true</upgradeOngoing>

Environment

• VMware vCenter Server 6.x
• VMware vCenter Server 7.0.x
• VMware VCenter Server 8.0.x

Cause

• This issue can happen if the vCenter had a failed upgrade(probably during import stage) but we continued to use that vCenter instead of reverting to a snapshot or backup and retry upgrade.
• That can be verified from /var/log/vmware/upgrade logs.

Resolution

Take a backup/snapshot of the vCenter before proceeding with the steps below :

1. Open a SSH session to the vCenter.

2. Go to /usr/lib/vmware-vpx/sps/conf/

3. Open sms-config.xml file using a file editor (vi).

4. Look for the following tag :

   <!-- To report the current upgrade state. In fresh install, this will always be false.

   Note : This is a system controlled parameter and should not be manually edited -->

   <upgradeOngoing>true</upgradeOngoing>

5. Change that value from true to false.

6. Restart SPS service: vmon-cli -r sps

7. Logout and login back to vCenter and you should be able to see Storage providers.

Additional Information

Impact/Risks:

• New VM creation can fail.
• Storage vMotion fails.
• vCLS deployment fails.