"A general system error occurred: PBM error occurred during PreReconfigureCallback: Invalid state" error when adding a disk to a Virtual Machine in the vCenter
search cancel

"A general system error occurred: PBM error occurred during PreReconfigureCallback: Invalid state" error when adding a disk to a Virtual Machine in the vCenter

book

Article ID: 424319

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • When attempting to add a disk to the Virtual Machine, the task fails with the following error:

    Operation failed!
    Task name         Apply Storage DRS recommendations
    Target  <VM NAME>
    Status   A general system error occurred: PBM error occurred during PreReconfigureCallback: Invalid state

  • The task is successful when triggered via ESXi Host Client.
  • Following errors are observed in the vCenter server:

    • /var/log/vmware/vmware-sps/sps.log
      YYYY-MM-DDTHH:MM:SS [main] WARN  opId=sps-Main-#####-### com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - loginByToken request timedout, cancelling the task scheduled
      YYYY-MM-DDTHH:MM:SS [main] ERROR opId=sps-Main-#####-### com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -
      com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service
      at com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException.fromEx(VpxdException.java:53) ~[storage-commons-1.0.jar:?]
              at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:175) ~[storage-commons-1.0.jar:?]
              at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:82) ~[storage-commons-1.0.jar:?]

    • /var/log/vmware/vpxd/vpxd.log
      YYYY-MM-DDTHH:MM:SS error vpxd[07091] [Originator@6876 sub=Default opID=#######-#######-#####] [VpxLRO] -- ERROR lro-13392474 -- ######-#####-#####-######(######-#####-#####-######) -- StorageResourceManager -- vim.StorageResourceManager.applyRecommendation: :vmodl.fault.SystemError
      --> Result:
      --> (vmodl.fault.SystemError) {
      -->    faultCause = (vmodl.MethodFault) null,
      -->    faultMessage = <unset>,
      -->    reason = " PBM error occurred during PreReconfigureCallback: Invalid state"
      -->    msg = ""


    • vmware-sps service is in a stopped state and does not start manually:

      service-control --status --all

      Running:
       applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-hvc vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp
      Stopped:
       vmcam vmonapi vmware-imagebuilder vmware-rbd-watchdog vmware-sps vmware-vcha

Cause

vCenter journalctl logs confirm that the issue is caused by missing permissions for the user VSPHERE.LOCAL\sps-#######-####-####-########.

Month date HH:MM:02 vcenter_fqdn vpxd[6346]: Event [52817928] [1-1] [YYYY-MM-DDTHH:MM:02.765692Z] [vim.event.NoAccessUserEvent] [error] [VSPHERE.LOCAL\sps-#######-####-####-########] [] [52817928] [Cannot login user VSPHERE.LOCAL\sps-#######-####-####-########@127.0.0.1: no permission]
Month date HH:MM:34 vcenter_fqdn vpxd[6346]: Event [52817937] [1-1] [YYYY-MM-DDTHH:MM:34.798338Z] [vim.event.NoAccessUserEvent] [error] [VSPHERE.LOCAL\sps-#######-####-####-########] [] [52817937] [Cannot login user VSPHERE.LOCAL\sps-#######-####-####-########@127.0.0.1: no permission]

Resolution

Ensure to take a snapshot of the vCenter before proceeding with the steps below. Take offline snapshots for all the vCenters in ELM mode.

To resolve this, add in the missing permission for the impacted sps user.

  1. Locate the user that has a missing permission in the journalctl logs.

    Example: VSPHERE.LOCAL\sps-######-####-####-####-######

  2. Go to the vCenter web UI client and login as [email protected]

    Example: [email protected]

  3. Click on the Root object of the VCSA
  4. Click on the Permissions tab
  5. Click Add
  6. Search for the string of the user that the vCenter is looking for
  7. Check the "Propagate to children" option and select Administrator role
  8. Save the change
  9. Restart the sps service:

    vmon-cli -r sps

  10. Confirm the vmware-sps service is now running with the command: service-control --status --all
 
Note: If the above steps results in an error within the UI, proceed with the workaround.


Workaround

Re-add the missing service account to the Administrators group.

  1. SSH to the affected vCenter via root
  2. Add the missing service account to the Administrators group:

    /usr/lib/vmware-vmafd/bin/dir-cli group modify --name Administrators --add sps-######-####-####-####-######

  3. Restart the sps service:

    vmon-cli -r sps

  4. Confirm the vmware-sps service is now running with the command: service-control --status

Alternate Solution

Run the lsdoctor tool with "-u" option to recreate the solution users completely.

This will repair any solution user accounts with permissions issues.

Additional Information

Error: "vim.fault.NoPermission" when SPS service fails to start

 

Powered by Wolken Software