After updating vCenter from 8.0.1 to 8.0.2, the SPS service on the vCenter stops working
search cancel

After updating vCenter from 8.0.1 to 8.0.2, the SPS service on the vCenter stops working

book

Article ID: 301488

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This article will give help you eliminate issue with SPS service not starting while VC is upgraded from 8.0 u1 to 8.0 u2.

While upgrading vCenter from 8.0 u1 to 8.0 u2, SPS service fails to start. This happens while VC is in ELM mode. In this scenario, we had 3 VC's in ELM out of which we were facing issues with only 1 VC.

This is what we would see in SPS logs:

Caused by: java.util.concurrent.TimeoutException
    at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:102) ~[vlsi-core.jar:?]
    at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:169) ~[storage-commons-1.0.jar:?]
    ... 208 more
YYYY-MM-DDTHH:MM:SS.xxxZ [main] INFO opId=sps-Main-99887-138 com.vmware.vim.storage.common.util.OperationIdUtil - OperationID present in invoker thread, adding suffix and r
e-using it - sps-Main-99887-138-100435-669.
YYYY-MM-DDTHH:MM:SS.xxxZ  [main] INFO opId=sps-Main-99887-138 com.vmware.vim.storage.common.util.OperationIdUtil - OperationID present in invoker thread, adding suffix and r
e-using it - sps-Main-99887-138-100435-669.
YYYY-MM-DDTHH:MM:SS.xxxZ  [main] INFO opId=sps-Main-99887-138 com.vmware.vim.storage.common.util.OperationIdUtil - OperationID present in invoker thread, adding suffix and r
e-using it - sps-Main-99887-138-100435-669.
YYYY-MM-DDTHH:MM:SS.xxxZ  [main] WARN opId=sps-Main-99887-138 com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - loginByToken request timedout, cancellin
g the task scheduled
YYYY-MM-DDTHH:MM:SS.xxxZ  [main] ERROR opId=sps-Main-99887-138 com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -
com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service
    at com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException.fromEx(VpxdException.java:53) ~[storage-commons-1.0.jar:?]
    at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:175) ~[storage-commons-1.0.jar:?]
    at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:82) ~[storage-commons-1.0.jar:?]
    at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:23) ~[storage-commons-1.0.jar:?]
    at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask$CallableTemplate.call(ConnectionInitializationTask.java:118) ~[storage-commons-1.0.jar:?]
    at com.vmware.vim.storage.common.task.retry.CallableRetryDecorator.call(CallableRetryDecorator.java:64) ~[storage-commons-1.0.jar:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
    at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.initialize(VpxdClientManagerImpl.java:138) ~[storage-commons-1.0.jar:?]
    at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.checkForInitialization(VpxdClientManagerImpl.java:132) ~[storage-commons-1.0.jar:?]
    at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getServiceContentHelper(VpxdClientManagerImpl.java:192) ~[storage-commons-1.0.jar:?]
    at com.vmware.vim.storage.common.security.PermissionValidator.getRootFolderFromVc(PermissionValidator.java:386) ~[storage-commons-1.0.jar:?]
    at com.vmware.vim.storage.common.security.PermissionValidator.<init>(PermissionValidator.java:76) ~[storage-commons-1.0.jar:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_362]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_362]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_362]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_362]

 

In vpxd, we will find the SPS service account which is failing to login:

Note: We have changed the account name as it was customer's environment information to VCENTER.

YYYY-MM-DDTHH:MM:SS.xxxZ info vpxd[06166] [Originator@6876 sub=AuthorizeManager opID=sps-Main-98895-593-98962-346-13] [Auth]: User VCENTER\sps-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxx
YYYY-MM-DDTHH:MM:SS.xxxZ warning vpxd[06166] [Originator@6876 sub=AuthorizeManager opID=sps-Main-98895-593-98962-346-13] Refresh function is not configured.User data can't be added to scheduler.User name: VCENTER\sps-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxx
YYYY-MM-DDTHH:MM:SS.xxxZ warning vpxd[06166] [Originator@6876 sub=Vmomi opID=sps-Main-98895-593-98962-346-13] VMOMI activation LRO failed; <<5211bd1c-7264-8581-3e90-6b106a760ac5, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 43068'>>, SessionManager, vim.SessionManager.loginByToken, <vim.version.v8_0_2_0, internal, 8.0.2.0>, {stm: {<io_obj p:0x00007f9780020130, h:84, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 43068'>>, id: 6503, state(in/out): 3/1}, session: <5211bd1c-7264-8581-3e90-6b106a760ac5, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 43068'>>, req: {POST, /sdk}}>, N3Vim5Fault12NoPermission9ExceptionE(Fault cause: vim.fault.NoPermission
--> )
--> [context]zKq7AVECAQAAAEuUVQEYdnB4ZAAAxbVTbGlidm1hY29yZS5zbwAAUglDAIwxRACaSEuBZVUIAXZweGQAgWy5CQGBvIDDAYGNj8MBgZmawwGBK/fCAYGJm8IBgiRMMgFsaWJ2aW0tdHlwZXMuc28AgdlyYwID1cMbbGlidm1vbWkuc28AgUCdQwKBnQNiAoHvE2ICgRkrYQKBtP1hAgDmyzcA+SQ4AJPAUQSujgBsaWJwdGhyZWFkLnNvLjAABS/eD2xpYmMuc28uNgA=[/context]
YYYY-MM-DDTHH:MM:SS.xxxZ info vpxd[06166] [Originator@6876 sub=vpxLro opID=sps-Main-98895-593-98962-346-13] [VpxLRO] -- FINISH lro-74506
YYYY-MM-DDTHH:MM:SS.xxxZ error vpxd[06166] [Originator@6876 sub=Default opID=sps-Main-98895-593-98962-346-13] [VpxLRO] -- ERROR lro-74506 -- 5211bd1c-7264-8581-3e90-6b106a760ac5 -- SessionManager -- vim.SessionManager.loginByToken: :vim.fault.NoPermission
--> Result:
--> (vim.fault.NoPermission) {
-->  faultCause = (vmodl.MethodFault) null,
-->  faultMessage = <unset>,
-->  object = 'vim.Folder:xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxx:group-d1',
-->  privilegeId = "System.View",
-->  missingPrivileges = (vim.fault.NoPermission.EntityPrivileges) [
-->    (vim.fault.NoPermission.EntityPrivileges) {
-->     entity = 'vim.Folderxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxx:group-d1',
-->     privilegeIds = (string) [
-->       "System.View"
-->     ]
-->    }
-->  ]
-->  msg = ""
--> }
--> Args:
-->
--> Arg locale:
--> "en_US"

 

Environment

VMware vCenter Server 8.0

Cause

SPS service account removed from the Administrators group. The addition of the sps Service Account to “ServiceProvideUsers” group has been a recent change( present in 8.0U2). We have noticed that the issue happens when VC is in ELM and when one of the VC is upgraded to 8.0 u2 and other are not which creates replication conflicts occurring in vmdir post the patch update. Engineering is still working on root cause.

Resolution

Engineering is still working on resolution. 


Workaround:

The workaround understood is to re-add the Service account to the Administrators group. You should be able to find out the SPS service account as highlighted above in vpxd.log. Run below command as mentioned:

  1. Take offline snapshots of all the vCenters in ELM mode
  2. Take SSH for VC in question with root privileges and run below command : /usr/lib/vmware-vmafd/bin/dir-cli group modify --name Administrators --add sps-xx-xx-xx-xx
  3. Restart sps services



Powered by Wolken Software