Error: "An error occurred while starting service 'sps' VMware vSphere Profile-Driven Storage Service failed to start" Dring vCenter 7.0 Upgrade
search cancel

Error: "An error occurred while starting service 'sps' VMware vSphere Profile-Driven Storage Service failed to start" Dring vCenter 7.0 Upgrade

book

Article ID: 336095

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Upgrading to vCenter Server 7.0 fails with error:
An error occurred while starting service 'sps'
VMware vSphere Profile-Driven Storage Service failed to start
  • In /var/log/firstboot/firstbootStatus.json file:
    "failedSteps": "sms_spbm_firstboot",
  • In /var/log/firstboot/sms_spbm_firstboot.py_XXXXX_stderr.log:

    2020-04-21T16:13:32.135Z  ERROR starting sps rc: 1, stdout: , stderr: Start service request failed. Error: Operation timed out
    2020-04-21T16:13:32.135Z  Failed to start the service : {
        "detail": [
            {
                "id": "install.ciscommon.service.failstart",
                "translatable": "An error occurred while starting service '%(0)s'",
                "args": [
                    "sps"
                ],
                "localized": "An error occurred while starting service 'sps'"
            }
        ],
        "componentKey": null,
        "problemId": null,
        "resolution": null
    }

  • In /var/log/vmware/vmware-sps/sps.log:

    2020-04-21T16:01:49.707Z [Thread-10] ERROR opId=sps-Main-484890-690 com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller - Unable to unmarshal stream to object.
    org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:168)
    .....
        at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:253)
    Caused by: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
    ....
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:163)
        ... 11 more
    2020-04-21T16:01:49.720Z [main] ERROR opId=sps-Main-484890-690 com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory - Missing namespace definition detected for namespace prefix xsd
    2020-04-21T16:01:49.725Z [main] ERROR opId=sps-Main-484890-690 com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory - Missing namespace definition detected for namespace prefix xsd
    2020-04-21T16:01:49.725Z [main] ERROR opId=sps-Main-484890-690 com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory - Missing namespace definition detected for namespace prefix xsd
    2020-04-21T16:01:49.723Z [Thread-10] ERROR opId=sps-Main-484890-690 com.vmware.vim.storage.common.kv.LocalKvReaderWriter - Failed to get values for the key prefix '' from the provider '[Name:providerInfoKvProvider Optimistic locking:]'.
    com.vmware.vim.storage.common.kv.exceptions.KvValueUnmarshalException: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:104)
    ....
        at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:253)
    Caused by: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:168)
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:101)
        ... 10 more
    Caused by: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
    ...
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:163)
        ... 11 more
    2020-04-21T16:01:49.728Z [Thread-10] ERROR opId=sps-Main-484890-690 com.vmware.vim.sms.StorageManagerImpl - Exception while loading providers!
    com.vmware.vim.storage.common.kv.exceptions.KvOperationException: Failed to get values for the key prefix '' from the provider '[Name:providerInfoKvProvider Optimistic locking:]'.
        at com.vmware.vim.storage.common.kv.LocalKvReaderWriter.getByPrefix(LocalKvReaderWriter.java:223)
    ....
        at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:253)
    Caused by: com.vmware.vim.storage.common.kv.exceptions.KvValueUnmarshalException: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:104)
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.toValueObject(KvDefaultValueMarshaller.java:158)
        at com.vmware.vim.storage.common.kv.LocalKvReaderWriter.getByPrefix(LocalKvReaderWriter.java:201)
        ... 8 more
    Caused by: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:168)
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:101)
        ... 10 more
    Caused by: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = 5180556415689811114, local class serialVersionUID = 1
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
    ....
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:163)
        ... 11 more

Environment

VMware vCenter Server 7.0.x

Cause

This issue occurs when there are storage providers in statusFault = (sms.fault.ProviderSyncFailed) state.

Resolution

This issue is resolved in vCenter Server 7.0.0b, see Download Broadcom products and software

Workaround
To workaround this issue, resolve the storage provider statusfault in source vCenter:

  • Finding storage provider in statusFault = (sms.fault.ProviderSyncFailed)
    Un-register and re-sync storage provider:

Finding storage provider in statusFault = (sms.fault.ProviderSyncFailed)

  1. Connect to the source vCenter sms Managed Object Browser to list Storage Providers using below URL with a vCenter Administrator user credentials:
https://<vCenter FQDN/IP>/sms/mob?moid=storageManager&method=QueryProvider
  1. Click on the "Invoke Method" link and note down the list of providers
For Example:
  1. On a new browser tab use the below URL with a vCenter Administrator user credentials to show details of the Provider:
https://<vCenter FQDN/IP>/sms/mob?moid=<VASAPROVIDER-ID>&method=QueryProviderInfo

For Example:

https://vcenter.example.com/sms/mob?moid=vasaProvider-1&method=QueryProviderInfo
  1. Click on the "Invoke Method" link.
  2. From the output search (Ctrl+F) for statusFault,if the value is NOT null, note down the uid of the provider:
For example: 
uid = 0e178603-1877-4c7b-8719-a69a76f591c8
  1. Repeat step 3 to 5 for each provider which are not having a statusFault = null

Un-register and re-sync storage provider:

Caution: IO Filter providers will register back on "Synchronize Storage Provider" from vSphere client, Other storage providers require manual registration after unregister.

  1. On a new browser tab connect to below URL with a vCenter Administrator user credentials:
https://<vCenter FQDN/IP>/sms/mob/?moid=storageManager&method=UnregisterProvider_Task

For Example: 

https://vcenter.example.com/sms/mob/?moid=storageManager&method=UnregisterProvider_Task
 
  1. In the value field fill the uid of the provider that need to be unregistered
  2. Click on "Invoke Method" link, the result may look like below:
  1. Go to vCenter's vSphere Client -> select vCenter Server object -> Configure -> Storage Providers
 
  1. Use the Synchronize Storage Providers  option to register the IO Filter providers back
  2. Use the Add option to register other storage providers back
  3. Validate the statusFault value again following  Finding storage provider in statusFault = (sms.fault.ProviderSyncFailed)
  4. Upgrade to vCenter Server 7.0
If this issue persists, contact VMware Support and note this Article ID (78900) in the problem description.
Powered by Wolken Software