SPS service start failing with Error "VpxdException: Error while doing login to VPXD service" on vCenter Server
search cancel

SPS service start failing with Error "VpxdException: Error while doing login to VPXD service" on vCenter Server

book

Article ID: 320872

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • VMware vSphere Profile-Driven Storage Service and VMware Update Manager service failing to start
  • Permissions on the vCenter server object is also inaccessible on the web client 

sps.log:

<YYYY-MM-DDTHH:MM:SS> [main] INFO  opId=sps-Main-360774-610 com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - Service solution user token acquired successfully. Expiration time: Wed May 15 20:40:38 UTC 2019
<YYYY-MM-DDTHH:MM:SS> [main] INFO  opId=sps-Main-360774-610 com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - SPS solution user initialized successfully
<YYYY-MM-DDTHH:MM:SS> [pool-1-thread-1] WARN  opId=sps-Main-360774-610 com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler - Error while trying to do relogin
java.lang.IllegalStateException: Client initialization is not complete!
        at com.google.common.base.Preconditions.checkState(Preconditions.java:174)
        at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask.get(ConnectionInitializationTask.java:80)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getClient(VpxdClientManagerImpl.java:169)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getCurrentSession(VpxdClientManagerImpl.java:180)
        at com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler.retry(RequestRetryHandler.java:76)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.setException(MethodInvocationHandlerImpl.java:525)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
<YYYY-MM-DDTHH:MM:SS> [main] ERROR opId=sps-Main-360774-610 com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - VPXD client login failed.
<YYYY-MM-DDTHH:MM:SS> [main] ERROR opId=sps-Main-360774-610 com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -
com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service
        at com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException.fromEx(VpxdException.java:53)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:159)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:129)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:34)
        at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask$CallableTemplate.call(ConnectionInitializationTask.java:118)
        at com.vmware.vim.storage.common.task.retry.CallableRetryDecorator.call(CallableRetryDecorator.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.initialize(VpxdClientManagerImpl.java:106)
        at com.vmware.sps.StorageMain.commonInitialization(StorageMain.java:188)
        at com.vmware.sps.StorageMain.main(StorageMain.java:67)
Caused by: (vim.fault.NoPermission) {
   faultCause = null,
   faultMessage = null,
   object = ManagedObjectReference: type = Folder, value = group-d1, serverGuid = ########-####-####-####-########1f07,
   privilegeId = System.View
}

vmware-vum-server-1.log:

<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,92] Ufa shutdown manager stopping... first make sure this thread is enlisted.
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=ThreadPool] Thread enlisted
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,97] Ufa shutdown manager stopping... Total number of callbacks: 1
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=VcIntegrity] Error on logout (ignored): Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem,  timeout, or service overload.
<YYYY-MM-DDTHH:MM:SS> warning vmware-vum-server[21223] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fa4dc03d230, h:15, <TCP '127.0.0.1 : 50448'>, <TCP '127.0.0.1 : 80'>>, e: 111(Connection refused)
<YYYY-MM-DDTHH:MM:SS> warning vmware-vum-server[21162] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fa5105085c0, h:15, <TCP '127.0.0.1 : 50450'>, <TCP '127.0.0.1 : 80'>>, e: 111(Connection refused)
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[21220] [Originator@6876 sub=VcIntegrity] Error on logout (ignored): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,109] Ufa shutdown manager stopped
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[21166] [Originator@6876 sub=ThreadPool] Thread delisted

vpxd.log:

<YYYY-MM-DDTHH:MM:SS> verbose vpxd[35323] [Originator@6876 sub=[SSO] opID=sps-Main-226767-398-34] [UserDirectorySso] NormalizeUserName(<SSO Domain>\vpxd-extension-########-####-####-####-########aa44, false) res: <SSO Domain>\vpxd-extension-########-####-####-####-########aa44
<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=AuthorizeManager opID=sps-Main-226767-398-34] [Auth]: User <SSO Domain>\vpxd-extension-########-####-####-####-########aa44
<YYYY-MM-DDTHH:MM:SS> verbose vpxd[35323] [Originator@6876 sub=Vmomi opID=sps-Main-226767-398-34] Invoke error: vim.SessionManager.loginByToken session: ########-####-####-####-########dfe6 Throw: vim.fault.NoPermission
<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=vpxLro opID=sps-Main-226767-398-34] [VpxLRO] -- FINISH lro-40
<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=Default opID=sps-Main-226767-398-34] [VpxLRO] -- ERROR lro-40 -- SessionManager -- vim.SessionManager.loginByToken: vim.fault.NoPermission:
--> Result:
--> (vim.fault.NoPermission) {
--> faultCause = (vmodl.MethodFault) null

Environment

VMware vCenter Server Appliance 6.7.x

Cause

The issue is seen if there are duplicate entries for the default local SSO administrator in VCDB

Resolution

This issue is resolved in vCenter Server 6.7 Update 3l build 17138064 and 7.0 Update 3c build 19234570 or later

Workaround:

** Ensure to take a functional backup of the nodes **

* Overview of Backup and Restore options in vCenter Server 6.x (2149237)
Backup and Restore options in vCenter Server 6.x/7.0.x/8.0.x - Overview

  • log in to the VCSA using ssh
  • stop the service using the following command 

service-control --stop vmware-vpxd
service-control --stop vmware-content-library

  • log in to the database using the following command:

command: /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

  • Execute the following steps to validate and remove the duplicate entries

VCDB=# select * from vpx_access;

Sample:
id | principal | role_id | entity_id | flag | surr_key
-----+---------------------------+---------+-----------+------+----------
1 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 1
8 | SSO.VCENTER\Administrator | -1 | 7 | 1 | 2
901 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 11

  • Additionally validate the entries where principal value is <SSO Domain>\Administrator

VCDB=# select * from vpx_access where principal like '%Administrator%';

 id  |          principal          | role_id | entity_id | flag | surr_key
-----+-----------------------------+---------+-----------+------+----------
   1 | SSO.VCENTER\Administrator |      -1 |         1 |    1 |        1
 209 | SSO.VCENTER\Administrator |      -1 |         2 |    1 |        3
 601 | SSO.VCENTER\Administrator |      -1 |         1 |    1 |        7

  • Delete the entries except id = 1

command: delete from vpx_access where id=<ID from above output>;
Note: command needs to be executed for all the additional entries

  • start the service using the following command:

service-control --start vmware-vpxd
service-control --start vmware-content-library
service-control --start vmware-sps
service-control --start vmware-updatemgr

  • Log in to the web client and validate the functionality



Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ

Impact/Risks:
Unable to add permissions on the vCenter server objects

Powered by Wolken Software