"An error occurred while starting service 'sps' VMware vSphere Profile-Driven Storage Service failed to start" During vCenter 7.0 Upgrade

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Upgrading to vCenter Server 7.0 Fails with Error:

An error occurred while starting service 'sps'
VMware vSphere Profile-Driven Storage Service failed to start

• In /var/log/firstboot/firstbootStatus.json file, you see entries similar to:

    "failedSteps": "sms_spbm_firstboot",

• In /var/log/firstboot/sms_spbm_firstboot.py_XXXXX_stderr.log, you see entries similar to:

2020-04-21T16:13:32.135Z  ERROR starting sps rc: 1, stdout: , stderr: Start service request failed. Error: Operation timed out

2020-04-21T16:13:32.135Z  Failed to start the service : {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "sps"
            ],
            "localized": "An error occurred while starting service 'sps'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

• In /var/log/vmware/vmware-sps/sps.log, you see entries similar to:

2020-04-21T16:01:49.707Z [Thread-10] ERROR opId=sps-Main-484890-690 com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller - Unable to unmarshal stream to object.
org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
	at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:168)
.....
	at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:253)
Caused by: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
....
	at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:163)
	... 11 more
2020-04-21T16:01:49.720Z [main] ERROR opId=sps-Main-484890-690 com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory - Missing namespace definition detected for namespace prefix xsd
2020-04-21T16:01:49.725Z [main] ERROR opId=sps-Main-484890-690 com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory - Missing namespace definition detected for namespace prefix xsd
2020-04-21T16:01:49.725Z [main] ERROR opId=sps-Main-484890-690 com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory - Missing namespace definition detected for namespace prefix xsd
2020-04-21T16:01:49.723Z [Thread-10] ERROR opId=sps-Main-484890-690 com.vmware.vim.storage.common.kv.LocalKvReaderWriter - Failed to get values for the key prefix '' from the provider '[Name:providerInfoKvProvider Optimistic locking:]'.
com.vmware.vim.storage.common.kv.exceptions.KvValueUnmarshalException: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
	at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:104)
....
	at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:253)
Caused by: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
	at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:168)
	at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:101)
	... 10 more
Caused by: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
...
	at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:163)
	... 11 more
2020-04-21T16:01:49.728Z [Thread-10] ERROR opId=sps-Main-484890-690 com.vmware.vim.sms.StorageManagerImpl - Exception while loading providers!
com.vmware.vim.storage.common.kv.exceptions.KvOperationException: Failed to get values for the key prefix '' from the provider '[Name:providerInfoKvProvider Optimistic locking:]'.
	at com.vmware.vim.storage.common.kv.LocalKvReaderWriter.getByPrefix(LocalKvReaderWriter.java:223)
....
	at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:253)
Caused by: com.vmware.vim.storage.common.kv.exceptions.KvValueUnmarshalException: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
	at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:104)
	at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.toValueObject(KvDefaultValueMarshaller.java:158)
	at com.vmware.vim.storage.common.kv.LocalKvReaderWriter.getByPrefix(LocalKvReaderWriter.java:201)
	... 8 more
Caused by: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
	at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:168)
	at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:101)
	... 10 more
Caused by: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
....
	at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:163)
	... 11 more

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 7.0.x

Cause

This issue occurs when there are storage providers in statusFault = (sms.fault.ProviderSyncFailed) state.

Resolution

This issue is resolved in vCenter Server 7.0.0b, available at VMware Downloads

Workaround:
To workaround this issue, resolve the storage provider statusfault in source vCenter:

• Finding storage provider in statusFault = (sms.fault.ProviderSyncFailed)
• Un-register and re-sync storage provider:

Finding storage provider in statusFault = (sms.fault.ProviderSyncFailed)

1. Connect to the source vCenter sms Managed Object Browser to list Storage Providers using below URL with a vCenter Administrator user credentials:

https://<vCenter FQDN/IP>/sms/mob?moid=storageManager&method=QueryProvider

2. Click on the "Invoke Method" link and note down the list of providers

For Example:

3. On a new browser tab use the below URL with a vCenter Administrator user credentials to show details of the Provider:

https://<vCenter FQDN/IP>/sms/mob?moid=<VASAPROVIDER-ID>&method=QueryProviderInfo

For Example:

https://vcenter.test.org/sms/mob?moid=vasaProvider-1&method=QueryProviderInfo

4. Click on the "Invoke Method" link.
5. From the output search (Ctrl+F) for statusFault, if the value is NOT null, note down the uid of the provider

For example: uid = 0e178603-1877-4c7b-8719-a69a76f591c8

6. Repeat step 3 to 5 for each provider which are not having a statusFault = null

Un-register and re-sync storage provider:

Caution: IO Filter providers will register back on "Synchronize Storage Provider" from vSphere client, Other storage providers require manual registration after unregister.

1. On a new browser tab connect to below URL with a vCenter Administrator user credentials:

https://<vCenter FQDN/IP>/sms/mob/?moid=storageManager&method=UnregisterProvider_Task

For Example: 

https://vcenter.test.org/sms/mob/?moid=storageManager&method=UnregisterProvider_Task
 

2. In the value field fill the uid of the provider that need to be unregistered
3. Click on "Invoke Method" link, the result may look like below:

4. Go to vCenter's vSphere Client -> select vCenter Server object -> Configure -> Storage Providers

 

5. Use the Synchronize Storage Providers  option to register the IO Filter providers back
6. Use the Add option to register other storage providers back
7. Validate the statusFault value again following  Finding storage provider in statusFault = (sms.fault.ProviderSyncFailed)
8. Upgrade to vCenter Server 7.0

If this issue persists, contact VMware Support and note this Article ID (78900) in the problem description. For more information, see How to Submit a Support Request.