"An error occurred while starting service 'sps' "- while upgrading vCenter Server to 6.7.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
Upgrading vCenter Server 6.7 fails and you see the errors:

• "An error occurred while starting service 'sps'"

The sms_spbm_firstboot.stderr.log (located in the var/log/firstboot/sms_spbm_firstboot.pystderr.log) contains an entry similar to:

2019-01-17T01:13:03.325Z  ERROR starting sps rc: 1, stdout: , stderr: Start service request failed. Error: Operation timed out
2019-01-17T01:13:03.326Z  Failed to start the service : {
    "componentKey": null,
    "resolution": null,
    "problemId": null,
    "detail": [
        {
            "args": [
                "sps"
            ],
            "localized": "An error occurred while starting service 'sps'",
            "translatable": "An error occurred while starting service '%(0)s'",
            "id": "install.ciscommon.service.failstart"

 

• The sps.log you see the below errors (located here var/log/vmware/vmware-sps)

2019-01-17T01:01:09.027Z [main] INFO  opId=sps-Main-686863-743 com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - SPS solution user initialized successfully
2019-01-17T01:01:09.419Z [pool-1-thread-1] WARN  opId=sps-Main-686863-743 com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler - Error while trying to do relogin
java.lang.IllegalStateException: Client initialization is not complete!
        at com.google.common.base.Preconditions.checkState(Preconditions.java:174)
        at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask.get(ConnectionInitializationTask.java:80)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getClient(VpxdClientManagerImpl.java:169)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getCurrentSession(VpxdClientManagerImpl.java:180)
        at com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler.retry(RequestRetryHandler.java:76)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.setException(MethodInvocationHandlerImpl.java:525)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-01-17T01:01:09.421Z [main] ERROR opId=sps-Main-686863-743 com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - VPXD client login failed.
2019-01-17T01:01:09.421Z [main] ERROR opId=sps-Main-686863-743 com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -
com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service

• The vpxd.log you see the below errors (located here var/log/vmware/vpxd)

2019-01-17T01:01:09.411Z info vpxd[22565] [Originator@6876 sub=AuthorizeManager opID=sps-Main-686863-743-3e] [Auth]: User VSPHERE.LOCAL\vpxd-extension-a1c69241-e10c-11e4-8619-000c
298c6105
2019-01-17T01:01:09.412Z info vpxd[22565] [Originator@6876 sub=vpxLro opID=sps-Main-686863-743-3e] [VpxLRO] -- FINISH lro-1010
2019-01-17T01:01:09.412Z info vpxd[22565] [Originator@6876 sub=Default opID=sps-Main-686863-743-3e] [VpxLRO] -- ERROR lro-1010 -- SessionManager -- vim.SessionManager.loginByToken
: vim.fault.NoPermission:
--> Result:
--> (vim.fault.NoPermission) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    object = 'vim.Folder:1C62C854-20F3-4B5E-BBDB-CF09776BF76A:group-d1',
-->    privilegeId = "System.View"
-->    msg = ""
--> }
--> Args:
-->
--> Arg locale:
--> "en_US"

Environment

VMware vCenter Server 6.7.x

Cause

The error occurs due to the permission for vpxd-extension been removed during the first sync with Authz.

Resolution

This issue has been resolved in vCenter Server 6.7 U1b/EP6, you can download the vCenter Server 6.7U1b VMware Downloads.