Error: "vim.fault.NoPermission" when SPS service fails to start
search cancel

Error: "vim.fault.NoPermission" when SPS service fails to start

book

Article ID: 323292

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Unable to create VMs on vSAN and VMFS datastores with error message "Cannot connect to profile-driven storage service"
  • vmware-sps service is stopped and does not start manually
  • vCenter - /var/log/vmware/vmware-sps/sps.log

[main] WARN opId=sps-Main- com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - loginByToken request timedout, cancelling the task scheduled ##.##.##.## [main] ERROR opId=sps-Main- com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception - com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service

  • Command "service-control --start vmware-sps" fails after some time
  • vCenter - /var/log/vmware/vmware-sps/sps.log 

Caused by: java.util.concurrent.TimeoutException  at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:102) ~[vlsi-core.jar:?]   at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:169) ~[storage-commons-1.0.jar:?]  ... 208 more

[main] INFO opId=sps-Main- com.vmware.vim.storage.common.util.OperationIdUtil - OperationID present in invoker thread, adding suffix and re-using it - sps-Main-

 [main] INFO opId=sps-Main- com.vmware.vim.storage.common.util.OperationIdUtil - OperationID present in invoker thread, adding suffix and re-using it - sps-Main-

 [main] INFO opId=sps-Main- com.vmware.vim.storage.common.util.OperationIdUtil - OperationID present in invoker thread, adding suffix and re-using it - sps-Main-

[pool-3-thread-11] INFO opId=sps-Main- com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Active thread count is: 11, Core Pool size is: 20, Queue size: 0, Time spent waiting in queue: 1 millis

[jaeger.RemoteReporter-QueueProcessor] WARN opId=sps-Main- io.jaegertracing.internal.reporters

.RemoteReporter - FlushCommand execution failed! Repeated errors of this command will not be logged.

io.jaegertracing.internal.exceptions.SenderException: Failed to flush spans.

       at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:116) ~[jaeger-thrift-1.8.0.jar:1.8.0]   at io.jaegertracing.internal.reporters.RemoteReporter$FlushCommand.execute(RemoteReporter.java:158) ~[jaeger-core-1.8.0.jar:1.8.0]  at io.jaegertracing.internal.reporters.RemoteReporter$QueueProcessor.run(RemoteReporter.java:179) [jaeger-core-1.8.0.jar:1.8.0] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]

Caused by: io.jaegertracing.internal.exceptions.SenderException: Could not send 2 spans  at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:86) ~[jaeger-thrift-1.8.0.jar:1.8.0]  at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[jaeger-thrift-1.8.0.jar:1.8.0]  ... 3 more
Caused by
...org.apache.thrift.transport.TTransportException: Cannot flush closed transport at io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:151) ~[jaeger-thrift-1.8.0.jar:1.8.0]  at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[libthrift-0.14.1.jar:0.14.1] at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) ~[libthrift-0.14.1.jar:0.14.1]  at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) ~[jaeger-thrift-1.8.0.jar:1.8.0]   at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[jaeger-thrift-.8.0.jar:1.8.0] ... 3 more 

  • vCenter - /var/log/vmware/vpxd.log

    info vpxd[05694] [Originator@6876 sub=vpxLro opID=sps-Main-] [VpxLRO] -- BEGIN lro-5678 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- ########-####-####-####-############
    info vpxd[05694] [Originator@6876 sub=vpxLro opID=sps-Main-] [VpxLRO] -- FINISH lro-5678
    info vpxd[05686] [Originator@6876 sub=vpxLro opID=sps-Main-] [VpxLRO] -- BEGIN lro-5679 -- ServiceInstance -- vim.ServiceInstance.retrieveInternalContent -- 
    info vpxd[05686] [Originator@6876 sub=vpxLro opID=sps-Main-] [VpxLRO] -- FINISH lro-5679
    info vpxd[05623] [Originator@6876 sub=vpxLro opID=sps-Main-] [VpxLRO] -- BEGIN lro-5680 -- SessionManager -- vim.SessionManager.loginByToken -- ########-####-####-####-#########
    info vpxd[05623] [Originator@6876 sub=UserDirectorySso opID=sps-Main-] GetUserInfoInternal(VSPHERE.LOCAL\sps-######-####-####-####-######, false) res: VSPHERE.LOCAL\sps-######-####-####-####-######
    info vpxd[05623] [Originator@6876 sub=AuthorizeManager opID=sps-Main-] [Auth]: User VSPHERE.LOCAL\sps-######-####-####-####-######
    warning vpxd[05623] [Originator@6876 sub=AuthorizeManager opID=sps-Main-] Refresh function is not configured.User data can't be added to scheduler.User name: VSPHERE.LOCAL\sps-######-####-####-####-######
    warning vpxd[05623] [Originator@6876 sub=Vmomi opID=sps-Main-] VMOMI activation LRO failed; <<######-####-####-####-######, <TCP '#.#.#.# : 8085'>, <TCP '#.#.#.# : 47688'>>, SessionManager, vim.SessionManager.loginByToken, <vim.version.v8_0_1_0, internal, 8.0.1.0>, {stm: {<io_obj p:0x00007f3b8400c1b0, h:91, <TCP '127.0.0.1 : 8085'>, <TCP '#.#.#.# : 47688'>>, id: 105, state(in/out): 3/1}, session: <######-####-####-####-######, <TCP '#.#.#.# : 8085'>, <TCP '#.#.#.# : 47688'>>, req: {POST, /sdk}}>, N3Vim5Fault12NoPermission9ExceptionE(Fault cause: vim.fault.NoPermission
    info vpxd[05623] [Originator@6876 sub=vpxLro opID=sps-Main-######-####-####-####-######] [VpxLRO] -- FINISH lro-5680
    error vpxd[05623] [Originator@6876 sub=Default opID=sps-Main-######-####-####-####-######] [VpxLRO] -- ERROR lro-5680 -- ######-####-####-####-###### -- SessionManager -- vim.SessionManager.loginByToken: :vim.fault.NoPermission
    ----
    warning vpxd[05623] [Originator@6876 sub=Vmomi opID=sps-Main-] VMOMI activation L
    RO failed; <<######-####-####-####-######, <TCP '#.#.#.# : 8085'>, <TCP '#.#.#.# : 47688'>>, SessionManager, vim.Sessi
    onManager.loginByToken, <vim.version.v8_0_1_0, internal, 8.0.1.0>, {stm: {<io_obj p:0x00007f3b8400c1b0, h:91, <TCP '#.#.#.# : 80
    85'>, <TCP '#.#.#.# : 47688'>>, id: 105, state(in/out): 3/1}, session: <######-####-####-####-######, <TCP '#.#.#.# : 
    8085'>, <TCP '#.#.#.# : 47688'>>, req: {POST, /sdk}}>, N3Vim5Fault12NoPermission9ExceptionE(Fault cause: vim.fault.NoPermission
    --> )
    --> [context]zKq7AVECAQAAAKD8SAEYdnB4ZAAA0cdUbGlidm1hY29yZS5zbwAAYQpGAFEGRwB1zUyBhWkPAXZweGQAgcyHGgGBW464AYERp7gBgeuyuAGBTxu4AYGE7
    rcBgps0RQFsaWJ2aW0tdHlwZXMuc28AgQjVSwIDrRMdbGlidm1vbWkuc28AgZSZLQKBGqVKAoFitUoCgdvdSQKBn4tKAgA5mjsAEuo7AOK2UgSHfwBsaWJwdGhyZWFkLnN
    vLjAABS82D2xpYmMuc28uNgA=[/context]
    info vpxd[05623] [Originator@6876 sub=vpxLro opID=sps-Main-######-####-####-####-######] [VpxLRO] -- FINISH l
    ro-5680
    error vpxd[05623] [Originator@6876 sub=Default opID=sps-Main-######-####-####-####-######] [VpxLRO] -- ERROR 
    lro-5680 -- ######-####-####-####-###### -- SessionManager -- vim.SessionManager.loginByToken: :vim.fault.NoPermission
    --> Result:
    --> (vim.fault.NoPermission) {
    -->    faultCause = (vmodl.MethodFault) null, 
    -->    faultMessage = <unset>, 
    -->    object = 'vim.Folder:#-#-#-#-#:group-d1', 
    -->    privilegeId = "System.View", 
    -->    missingPrivileges = (vim.fault.NoPermission.EntityPrivileges) [
    -->       (vim.fault.NoPermission.EntityPrivileges) {
    -->          entity = 'vim.Folder:#-#-#-#:group-d1', 
    -->          privilegeIds = (string) [
    -->             "System.View"
    -->          ]
    -->       }
    -->    ]
    -->    msg = ""
    --> }
    --> Args:
    --> 
    --> Arg locale:
    --> "en_US"

Environment

VMware vCenter Server 7.0.X
VMware vCenter Server 8.0.X

Cause

There is a missing permission for the vpxd-svcs solution user in VCSA. For example, in vpxd.log, vpxd is looking for VSPHERE.LOCAL\sps-######-####-####-####-######.

Resolution

Ensure to take a snapshot of the vCenter before proceeding with the steps below.

To resolve this, add in the missing permission for the noted user.

  1. Locate the user that has a missing permission in the vpxd.log.

    Example: VSPHERE.LOCAL\sps-######-####-####-####-######

  2. Go to the vCenter web UI client and login as [email protected]

    Example: [email protected]

  3. Click on the Root object of the VCSA
  4. Click on the Permissions tab
  5. Click Add
  6. Search for the string of the user that the vCenter is looking for
  7. Check the "Propagate to children" option and select Administrator role
  8. Save the change
  9. Restart the sps service:

    vmon-cli -r sps
  10. Confirm the vmware-sps service is now running with the command: service-control --status
 
Note: If the above steps results in an error within the UI, proceed with the workaround.
 

Workaround

Re-add the missing service account to the Administrators group.

  1. Take offline snapshots of all the vCenters in ELM mode
  2. SSH to the affected vCenter via root
  3. Add the missing service account to the Administrators group:

    /usr/lib/vmware-vmafd/bin/dir-cli group modify --name Administrators --add sps-######-####-####-####-######

  4. Restart the sps service:

    vmon-cli -r sps


  5. Confirm the vmware-sps service is now running with the command: service-control --status

 

Alternate Solution

Run the lsdoctor tool with "-u" option to recreate the solution users completely.

This will repair any solution user accounts with permissions issues.

Additional Information

Powered by Wolken Software