Storage Providers show vSAN plugin in error state causing external replication solutions fail to initialize

Products

VMware vCenter Server

Issue/Introduction

When managing your vCenter Server environment, you may observe that the built-in vSAN storage provider (plugin) under Storage Providers displays an error state, or external disaster recovery and replication solutions (such as Dell RecoverPoint) become stuck in an "Initializing" or "Paused by System" status.

When this issue occurs, you will notice the following symptoms:

The vSAN plugin under Storage Providers in the vSphere Client user interface is displayed in an error state.
The underlying vsan-health service on the vCenter Server is in a stopped state or its startup configuration is set to a manual start type.
Automated reconfiguration tasks from external storage or replication service accounts continuously run and complete, but replication states do not progress.
The vSphere Profile-Driven Storage service logs (sps.log) report HTTP 503 errors or VasaServiceException faults when trying to fetch storage profile association information or queryAssociatedEntity API tasks

The following trace is recorded in vCenter sps logs:

/var/log/vmware/vmware-sps/sps.log:

YYYY-MM-DDTHH:MM:SS [pool-X-thread-Y] ERROR com.vmware.pbm.persistence.impl.AssociationPersistenceManagerImpl - Failed to fetch association information for profile (pbm.profile.ProfileId) {
dynamicType = null,
   dynamicProperty = null,
   uniqueId = <PROFILE_UUID>
} and entity type null
java.util.concurrent.ExecutionException: com.vmware.vim.sms.fault.VasaServiceException: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at com.vmware.pbm.persistence.impl.AssociationPersistenceManagerImpl.fetchFileShareEntityAssociationsConcurrently(AssociationPersistenceManagerImpl.java:297)
        at com.vmware.pbm.persistence.impl.AssociationPersistenceManagerImpl.getAssociationData(AssociationPersistenceManagerImpl.java:247)
        at com.vmware.pbm.profile.impl.AssociationServiceImpl.getEntityAssociationData(AssociationServiceImpl.java:314)
        at com.vmware.pbm.profile.impl.ProfileManagerImpl.queryAssociatedEntitiesInt(ProfileManagerImpl.java:4499)
        at com.vmware.pbm.profile.impl.ProfileManagerImpl.queryAssociatedEntityInt(ProfileManagerImpl.java:1893)
        at com.vmware.pbm.profile.impl.ProfileManagerImpl.queryAssociatedEntity(ProfileManagerImpl.java:1844)
        ...
Caused by: com.vmware.vim.sms.fault.VasaServiceException: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at com.vmware.vim.sms.client.vmomi.VasaVmomiServiceClient.createPolicyManager(VasaVmomiServiceClient.java:270)
        at com.vmware.vim.sms.client.vmomi.VasaVmomiServiceClient.getPolicyManager(VasaVmomiServiceClient.java:111)
        at com.vmware.vim.sms.client.vmomi.VmodlVasaClientImpl.getAssociatedObjectsV2(VmodlVasaClientImpl.java:373)
        ...

Review of the vsanvcmgmtd log records reveals exact transport disconnection timestamps when the remote peer connection is shut down

/var/log/vmware/vsan-health/vsanvcmgmtd.log:

YYYY-MM-DDTHH:MM:SS info vsanvcmgmtd[823245] [vSAN@6876 sub=Http2ClientSession-5 opId=Run-Http2ClientSession-5] Stream has been reset; {<io_obj p:0x00007fefd42765d0, h:-1, <TCP '127.0.0.1 : 54902'>, <TCP '127.0.0.1 : 8085'>>, id: 1272055, state(in/out): 1/3}, Transport connection closed by remote - read = 0
YYYY-MM-DDTHH:MM:SS error vsanvcmgmtd[824981] [vSAN@6876 sub=IO.Http opId=vsan-PC-651add3d8bc95] User agent failed to send request; {stm: {<io_obj p:0x00007fefd42765d0, h:-1, <TCP '127.0.0.1 : 54902'>, <TCP '127.0.0.1 : 8085'>>, id: 1272055, state(in/out): 4/4}, req: {OPTIONS, }}, N7Vmacore16Http2IOExceptionE(Connection was closed by the remote peer)
YYYY-MM-DDTHH:MM:SS warning vsanvcmgmtd[823400] [vSAN@6876 sub=Py2CppStub opId=vsan-PC-651add3d8bc95] Exception while invoking VMOMI method 'LOCAL::vmodl.query.PropertyCollector.waitForUpdatesEx': N7Vmacore16Http2IOExceptionE(Connection was closed by the remote peer)

Environment

VMware vCenter Server

Storage Policy Based Management (SPBM)

Cause

This issue occurs when the vSAN storage provider plugin is in error state while user tries to unregister/delete the plugin from vSphere-UI and/or if the vsan-health services is stopped.

The vsan-health service is the internal vCenter Server service responsible for tracking and communicating core datastore health capabilities. Irrespective of whether an environment explicitly utilizes an active vSAN datastore array or has deleted its storage definitions, the built-in vSAN storage provider architecture is natively integrated into the core vCenter Server management layer by design. Even in the absence of a vSAN datastore within the infrastructure, the internal vSAN Storage Provider is a core, mandated component of the default vCenter Server system design. It acts as an essential, baseline tracking layer for general Storage Policy Based Management (SPBM) and Profile-Driven Storage functions across all storage types. Because these native storage extensions are tightly integrated into the core vCenter engine, you cannot manually unregister certain storage providers like vSAN supplied by design.

The vSphere Profile-Driven Storage (SPS) service depends on continuous end-to-end communication with this internal management endpoint to resolve Storage Policy Based Management (SPBM) calls and handle object association queries. If the underlying vsan-health service is offline, the unified storage provider endpoint becomes unreachable, causing internal web components to respond with HTTP 503 (Service Unavailable) status codes breaking configuration tasks for both internal operations and third-party solutions.

Resolution

To resolve this issue, restore the service execution state and ensure configure it to start automatically. When the vsan-health service starts or restarts, it triggers an automated bootstrap process that bypasses the locked vCenter user interface to register the vSAN storage provider. It validates internal capabilities, serializes its VASA XML metadata, and calls the vCenter API directly to bind the provider endpoints into the active registry. Restarting the service completely restores this broken link.

Log in to the vCenter Server Appliance via SSH using root credentials.
Check the current runtime status and start type of the vSAN Health service by executing the following command:
```
vmon-cli -s vsan-health
```
If the service status output reveals that the service is in a STOPPED state and/or its startup configuration is configured as MANUAL, you must modify it to persist across lifecycles. Change the configuration start type of the service to AUTOMATIC by executing:
```
vmon-cli --update vsan-health --starttype AUTOMATIC
```
Start the service manually by running:
```
vmon-cli -r vsan-health
```
Starting the vsan-health service restores the endpoint infrastructure used by built-in storage software components to pass key-value datastore capabilities to the vCenter Server layer. This clears the unexpected HTTP 503 errors and allows PbmQueryAssociatedEntityCommand workflows triggered by external solutions like Dell RecoverPoint to map profile environments normally.
If the storage provider status does not immediately transition to online within the vSphere Client UI, restart the vSphere Profile-Driven Storage service:
```
vmon-cli -r sps
```
Navigate back to the vSphere Client, select your vCenter Server, go to the Configure tab, select Storage Providers, and click Resync to verify that the provider status returns to active and online.

Additional Information

Always ensure that built-in vCenter storage extensions are left intact and running, as third-party storage plugins and core compliance engines rely on the baseline VASA registration framework to map profile environments normally.

Refer: Manage Storage Providers and View vSAN Storage Providers for more information.