Error: "An error occurred while starting service 'sps' VMware vSphere Profile-Driven Storage Service failed to start" During vCenter 7.0 Upgrade
search cancel

Error: "An error occurred while starting service 'sps' VMware vSphere Profile-Driven Storage Service failed to start" During vCenter 7.0 Upgrade

book

Article ID: 336095

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Upgrading to vCenter Server 7.0 fails with error:
An error occurred while starting service 'sps'
VMware vSphere Profile-Driven Storage Service failed to start
  • In /var/log/firstboot/firstbootStatus.json file:
    "failedSteps": "sms_spbm_firstboot",
  • In /var/log/firstboot/sms_spbm_firstboot.py_#####_stderr.log:

    DD-MM-YYThh:mm:ss  ERROR starting sps rc: 1, stdout: , stderr: Start service request failed. Error: Operation timed out
    DD-MM-YYThh:mm:ss  Failed to start the service : {
        "detail": [
            {
                "id": "install.ciscommon.service.failstart",
                "translatable": "An error occurred while starting service '%(0)s'",
                "args": [
                    "sps"
                ],
                "localized": "An error occurred while starting service 'sps'"
            }
        ],
        "componentKey": null,
        "problemId": null,
        "resolution": null
    }

  • In /var/log/vmware/vmware-sps/sps.log:

    DD-MM-YYThh:mm:ss [Thread-10] ERROR opId=sps-Main-#####-690 com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller - Unable to unmarshal stream to object.
    org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = #####, local class serialVersionUID = 1
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:168)
    .....
        at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:253)
    Caused by: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = #####, local class serialVersionUID = 1
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
    ....
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:163)
        ... 11 more
    DD-MM-YYThh:mm:ss [main] ERROR opId=sps-Main-#####-690 com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory - Missing namespace definition detected for namespace prefix xsd
    DD-MM-YYThh:mm:ss [main] ERROR opId=sps-Main-#####-690 com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory - Missing namespace definition detected for namespace prefix xsd
    DD-MM-YYThh:mm:ss [main] ERROR opId=sps-Main-#####-690 com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory - Missing namespace definition detected for namespace prefix xsd
    DD-MM-YYThh:mm:ss [Thread-10] ERROR opId=sps-Main-#####-690 com.vmware.vim.storage.common.kv.LocalKvReaderWriter - Failed to get values for the key prefix '' from the provider '[Name:providerInfoKvProvider Optimistic locking:]'.
    com.vmware.vim.storage.common.kv.exceptions.KvValueUnmarshalException: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = #####, local class serialVersionUID = 1
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:104)
    ....
        at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:253)
    Caused by: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = #####, local class serialVersionUID = 1
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:168)
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:101)
        ... 10 more
    Caused by: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = #####, local class serialVersionUID = 1
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
    ...
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:163)
        ... 11 more
    DD-MM-YYThh:mm:ss [Thread-10] ERROR opId=sps-Main-#####-690 com.vmware.vim.sms.StorageManagerImpl - Exception while loading providers!
    com.vmware.vim.storage.common.kv.exceptions.KvOperationException: Failed to get values for the key prefix '' from the provider '[Name:providerInfoKvProvider Optimistic locking:]'.
        at com.vmware.vim.storage.common.kv.LocalKvReaderWriter.getByPrefix(LocalKvReaderWriter.java:223)
    ....
        at com.vmware.vim.sms.StorageManagerImpl$ProviderLoader.run(StorageManagerImpl.java:253)
    Caused by: com.vmware.vim.storage.common.kv.exceptions.KvValueUnmarshalException: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = #####, local class serialVersionUID = 1
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:104)
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.toValueObject(KvDefaultValueMarshaller.java:158)
        at com.vmware.vim.storage.common.kv.LocalKvReaderWriter.getByPrefix(LocalKvReaderWriter.java:201)
        ... 8 more
    Caused by: org.apache.commons.lang.SerializationException: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = #####, local class serialVersionUID = 1
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:168)
        at com.vmware.vim.storage.common.kv.KvDefaultValueMarshaller.unmarshal(KvDefaultValueMarshaller.java:101)
        ... 10 more
    Caused by: java.io.InvalidClassException: com.vmware.vim.binding.impl.vmodl.LocalizableMessageImpl; local class incompatible: stream classdesc serialVersionUID = #####, local class serialVersionUID = 1
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
    ....
        at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:163)
        ... 11 more

Environment

VMware vCenter Server 7.0.x

Cause

This issue occurs when there are storage providers in statusFault = (sms.fault.ProviderSyncFailed) state.

Resolution

This issue is resolved in vCenter Server 7.0.0b, see Download Broadcom products and software

To workaround this issue, resolve the storage provider statusfault in source vCenter:

  • Finding storage provider in statusFault = (sms.fault.ProviderSyncFailed).
  • Un-register and re-sync storage provider.

Finding storage provider in statusFault = (sms.fault.ProviderSyncFailed)

  1. Connect to the source vCenter sms Managed Object Browser to list Storage Providers using below URL with a vCenter "Administrator" user credentials:
https://<vCenter FQDN/IP>/sms/mob?moid=storageManager&method=QueryProvider
  1. Click on the "Invoke Method" link and note down the list of providers
For Example:
  1. On a new browser tab use the below URL with a vCenter Administrator user credentials to show details of the Provider:
https://<vCenter FQDN/IP>/sms/mob?moid=<VASAPROVIDER-ID>&method=QueryProviderInfo

Note: Replace <VASAPROVIDER-ID> with the actual ID returned in output from the previous step
  1. Click on the "Invoke Method" link.
  2. From the output search (Ctrl+F) for statusFault,if the value is NOT null, note down the uid of the provider:
For example: 
uid = ########-####-####-####-############
  1. Repeat step 3 to 5 for each provider which are not having a statusFault = null

Un-register and re-sync storage provider:

Caution: IO Filter providers will register back on "Synchronize Storage Provider" from vSphere client, Other storage providers require manual registration after unregister.

  1. On a new browser tab connect to below URL with a vCenter "Administrator" user credentials:
https://<vCenter FQDN/IP>/sms/mob/?moid=storageManager&method=UnregisterProvider_Task 
  1. In the value field fill the uid of the provider that need to be unregistered
  2. Click on "Invoke Method" link, the result may look like below:
  1. Go to vCenter > Select vCenter Server object > Configure > Storage Providers
 
  1. Use the Synchronize Storage Providers  option to register the IO Filter providers back
  2. Use the Add option to register other storage providers back
  3. Validate the statusFault value again following steps mentioned in "Finding storage provider in statusFault = (sms.fault.ProviderSyncFailed)"
  4. Upgrade to vCenter Server 7.0
If this issue persists, contact VMware Support and note this Article ID (336095) in the problem description.
Powered by Wolken Software