SPS service start failing with Error "VpxdException: Error while doing login to VPXD service" on vCenter Server
search cancel

SPS service start failing with Error "VpxdException: Error while doing login to VPXD service" on vCenter Server

book

Article ID: 320872

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • VMware vSphere Profile-Driven Storage Service and VMware Update Manager service are failing to start
  • Permissions on the vCenter server object are also inaccessible on the web client
  • Below entries can be seen in the vCenter logs:

    • /var/log/vmware/vmware-sps/sps.log:
<YYYY-MM-DDTHH:MM:SS> [main] INFO  opId=sps-Main-360774-610 com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - Service solution user token acquired successfully. Expiration time: Wed May <date><time>
<YYYY-MM-DDTHH:MM:SS> [main] INFO  opId=sps-Main-360774-610 com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - SPS solution user initialized successfully
<YYYY-MM-DDTHH:MM:SS> [pool-1-thread-1] WARN  opId=sps-Main-360774-610 com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler - Error while trying to do relogin
java.lang.IllegalStateException: Client initialization is not complete!
        at com.google.common.base.Preconditions.checkState(Preconditions.java:174)
        at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask.get(ConnectionInitializationTask.java:80)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getClient(VpxdClientManagerImpl.java:169)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getCurrentSession(VpxdClientManagerImpl.java:180)
        at com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler.retry(RequestRetryHandler.java:76)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.setException(MethodInvocationHandlerImpl.java:525)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
<YYYY-MM-DDTHH:MM:SS> [main] ERROR opId=sps-Main-360774-610 com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - VPXD client login failed.
<YYYY-MM-DDTHH:MM:SS> [main] ERROR opId=sps-Main-360774-610 com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -
com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service
        at com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException.fromEx(VpxdException.java:53)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:159)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:129)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:34)
        at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask$CallableTemplate.call(ConnectionInitializationTask.java:118)
        at com.vmware.vim.storage.common.task.retry.CallableRetryDecorator.call(CallableRetryDecorator.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.initialize(VpxdClientManagerImpl.java:106)
        at com.vmware.sps.StorageMain.commonInitialization(StorageMain.java:188)
        at com.vmware.sps.StorageMain.main(StorageMain.java:67)
Caused by: (vim.fault.NoPermission) {
   faultCause = null,
   faultMessage = null,
   object = ManagedObjectReference: type = Folder, value = group-d1, serverGuid = ########-####-####-####-########1f07,
   privilegeId = System.View
}
    • /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log:
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,92] Ufa shutdown manager stopping... first make sure this thread is enlisted.
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=ThreadPool] Thread enlisted
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,97] Ufa shutdown manager stopping... Total number of callbacks: 1
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=VcIntegrity] Error on logout (ignored): Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem,  timeout, or service overload.
<YYYY-MM-DDTHH:MM:SS> warning vmware-vum-server[21223] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fa4dc03d230, h:15, <TCP '127.0.0.1 : 50448'>, <TCP '127.0.0.1 : 80'>>, e: 111(Connection refused)
<YYYY-MM-DDTHH:MM:SS> warning vmware-vum-server[21162] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fa5105085c0, h:15, <TCP '127.0.0.1 : 50450'>, <TCP '127.0.0.1 : 80'>>, e: 111(Connection refused)
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[21220] [Originator@6876 sub=VcIntegrity] Error on logout (ignored): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,109] Ufa shutdown manager stopped
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[21166] [Originator@6876 sub=ThreadPool] Thread delisted
    • /var/log/vmware/vpxd/vpxd.log:

<YYYY-MM-DDTHH:MM:SS> verbose vpxd[35323] [Originator@6876 sub=[SSO] opID=sps-Main-226767-398-34] [UserDirectorySso] NormalizeUserName(<SSO Domain>\vpxd-extension-########-####-####-####-########aa44, false) res: <SSO Domain>\vpxd-extension-########-####-####-####-########aa44
<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=AuthorizeManager opID=sps-Main-226767-398-34] [Auth]: User <SSO Domain>\vpxd-extension-########-####-####-####-########aa44
<YYYY-MM-DDTHH:MM:SS> verbose vpxd[35323] [Originator@6876 sub=Vmomi opID=sps-Main-226767-398-34] Invoke error: vim.SessionManager.loginByToken session: ########-####-####-####-########dfe6 Throw: vim.fault.NoPermission
<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=vpxLro opID=sps-Main-226767-398-34] [VpxLRO] -- FINISH lro-40
<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=Default opID=sps-Main-226767-398-34] [VpxLRO] -- ERROR lro-40 -- SessionManager -- vim.SessionManager.loginByToken: vim.fault.NoPermission:
--> Result:
--> (vim.fault.NoPermission) {
--> faultCause = (vmodl.MethodFault) null

Environment

  • VMware vCenter Server Appliance 6.7
  • VMware vCenter Server 7.0
 

Cause

This issue occurs when duplicate entries exist for the default local SSO administrator in VCDB

Resolution

This issue is resolved in vCenter Server 6.7 Update 3l build 17138064 and 7.0 Update 3c build 19234570 or later

Workaround:

Note: Ensure to take a functional backup of the nodes before proceeding with the below steps. Reference KB: Backup and Restore options in vCenter Server 6.x/7.0.x/8.0.x - Overview

  1. Log in to the vCenter via SSH(putty).
  2. Stop the vpxd and content-library service using the below command:

    service-control --stop vmware-vpxd
    service-control --stop vmware-content-library

  3. Log in to the VCDB using the following command:

    /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

  4. Execute the following steps to validate and remove the duplicate entries

    VCDB=# select * from vpx_access;

    Sample:
    id | principal | role_id | entity_id | flag | surr_key
    -----+---------------------------+---------+-----------+------+----------
    1 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 1
    8 | SSO.VCENTER\Administrator | -1 | 7 | 1 | 2
    901 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 11
  5. Additionally validate the entries where principal value is <SSO Domain>\Administrator

    VCDB=# select * from vpx_access where principal like '%Administrator%';

     id  |          principal          | role_id | entity_id | flag | surr_key
    -----+-----------------------------+---------+-----------+------+----------
       1 | SSO.VCENTER\Administrator |      -1 |         1 |    1 |        1
     209 | SSO.VCENTER\Administrator |      -1 |         2 |    1 |        3
     601 | SSO.VCENTER\Administrator |      -1 |         1 |    1 |        7
  6. Delete the entries except for the entry where id = 1

    VCDB=# delete from vpx_access where id='<ID from above output>';

    Note: command needs to be executed for all the additional entries

  7. Start the service using the following command:

    service-control --start vmware-vpxd
    service-control --start vmware-content-library
    service-control --start vmware-sps
    service-control --start vmware-updatemgr

  8. Log in to the web client and validate the functionality

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ

Impact/Risks:
Unable to add permissions on the vCenter server objects

Powered by Wolken Software