"A general runtime error occured" while selecting storage for vmotion, while deploying OVF/OVA, or when creating a VM in vSphere 8.0.U2 with ELM setup

Products

• VMware vCenter Server

Issue/Introduction

Unable to select storage for vMotion, OVF/OVA deployment or VM creation

/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

[YYYY-MM-DDThh:mm:ss.244-06:00] [ERROR] p-nio-127.0.0.1-5090-exec-39 70000928 100008 200004 com.vmware.vsphere.client.spbm.impl.PbmMutationProvider Failed to execute the compatibility request. com.vmware.vim.binding.vmodl.RuntimeFault: null

at sun.reflect.GeneratedConstructorAccessor541.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at java.lang.Class.newInstance(Class.java:442)

at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174)

at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25)

at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30)

/var/log/vmware/vmware-sps/sps.log

YYYY-MM-DDThh:mm:ss.494-05:00 [pool-4-thread-8] ERROR opId=lnxj3k20-46359-auto-zru-h5:70008404 com.vmware.pbm.vapi.authorization.ProfilePermissionAPIValidatorImpl - [checkProfileValidity] Exception occurred during getEffectivePrivilege

com.vmware.vim.sso.admin.exception.NoPermissionException: null

 at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.execute(VmomiClientCommand.java:192) ~[sso-adminsdk.jar:?]

 at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.executeEnsuringDomainErrorIs(VmomiClientCommand.java:248) ~[sso-adminsdk.jar:?]

 at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.executeEnsuringDomainErrorIs(VmomiClientCommand.java:225) ~[sso-adminsdk.jar:?]

 at com.vmware.vim.sso.admin.client.vmomi.impl.DomainManagementImpl.getDomains(DomainManagementImpl.java:330) ~[sso-adminsdk.jar:?]

/var/log/vmware/vpxd/vpxd.log:

vpxd-158.log:53211:YYYY-MM-DDThh:mm:ss.999-06:00 info vpxd[05761] [Originator@6876 sub=User opID=sps-Main-644532-452-644535-826-e5] Login token: SamlToken [subject={Name: sps-91####0a-####-####-####-91########1d; Domain:VSPHERE.LOCAL}, groups=[{Name: SolutionUsers; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: ServiceProviderUsers; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=yyyy-mm-dd hh:mm:ss.884, expirationTime=YYYY:MM:DD hh:mm:ss.884, renewable=false, delegable=true, isSolution=false,confirmationType=1]

Or in the journal log messages (journalctl -b 0:)

journalctl_-b--0.txt:MM DD hh:mm:ss #########.####.com vpxd[5819]: Event [56484388] [1-1] [YYYY-MM-DDThh:mm:ss.764323Z] [vim.event.NoAccessUserEvent] [error] [VSPHERE.LOCAL\sps-########-####-####-####-############] [] [56484388] [Cannot login user VSPHERE.LOCAL\sps-########-####-####-####-############@127.0.0.1: no permission]

Environment

VMware vCenter Server 8.0

Cause

In an vCenter ELM setup, when the first vCenter is upgraded to the 8.0U2 target (which has the fix) and other vCenter partners in the ELM remain at 7.x release versions (which doesn't have the fix) will encounter this issue.

Resolution

 VMware Engineering is aware of this issue and is working on a permanent fix.

To resolve the issue, follow any one the below options.

Option 1:

To address this issue temporarily, re-add the respective service account to the necessary group. Using the SPS service account as an example, follow these steps:

1. Take offline snapshots of all vCenters in the ELM setup.
2. Access the vCenter in question via SSH with root privileges.
3. Run the following commands to check if SPS user is in all of them: 
    /usr/lib/vmware-vmafd/bin/dir-cli group list --name Administrators
    /usr/lib/vmware-vmafd/bin/dir-cli group list --name ServiceProviderUsers
    /usr/lib/vmware-vmafd/bin/dir-cli group list --name ActAsUsers

  For example, while running the: /usr/lib/vmware-vmafd/bin/dir-cli group list --name ServiceProviderUsers 
  You will get an output similar to: CN=sps-xx-xx-xx-xx-xx,cn=xyz,dc=vcenter,dc=xyz, Note the "sps-xx-xx-xx-xx-xx" section.
  The SPS user must be missing in the Administrators section.

4. Run the following commands to add the SPS service account to the Administrators group:
    /usr/lib/vmware-vmafd/bin/dir-cli group modify --name Administrators --add sps-xx-xx-xx-xx-xx
5. Restart the SPS services using: service-control --stop sps && service-control --start sps

Option 2:

1. Remove the cache of the sps user account: rm /var/cache/svcaccounts/sps/.sps
2. Restart the sps service:  vmon-cli --restart sps