VM cloning fails with "PBM error occurred during PreCloneCheckCommonCallback: Fault cause: vmodl.RuntimeFault"

Products

• VMware vCenter Server

Issue/Introduction

• You find that the only way to temporarily fix the issue is to reboot the vCenter.
• In the vCenter /var/log/vmware/vpxd.log you find that the clone task fails with the 'queryAssociatedProfiles' task:

info vpxd[06599] [Originator@6876 sub=vpxLro opID=1f6ebb52] [VpxLRO] -- BEGIN task-446739 -- vm-2021 -- vim.VirtualMachine.clone -- 520da1dd-944b-5a4e->7a9b-727a6ac4aa2f(5260b799-affb-9688-63ac-89f4334abbe9)

info vpxd[06599] [Originator@6876 sub=vmomi.soapStub[958] opID=1f6ebb52-01] SOAP request returned HTTP failure; <<cs p:00007f348c002760, TCP:localhost:1080>, /pbm/sdk>, method: queryAssociatedProfiles; code: 500(Internal Server Error); fault: (vmodl.RuntimeFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "GeneralError.summary",
-->          arg = <unset>,
-->          message = <unset>
-->       }
-->    ]
-->    msg = "Received SOAP response fault from [<<cs p:00007f348c002760, TCP:localhost:1080>, /pbm/sdk>]: queryAssociatedProfiles
--> "
--> }
error vpxd[06599] [Originator@6876 sub=pbm opID=1f6ebb52-01] PBM error occurred during PreCloneCheckCommonCallback: Fault cause: vmodl.RuntimeFault

• In the vCenter /var/log/vmware/sps.log you see that a 'vim.fault.NoPermission' error is reported when trying to read a property during the clone task:
  
  [pool-4-thread-19] WARN  opId=1f6ebb52-01 com.vmware.vim.storage.common.vc.impl.VcInventoryImpl - Error retrieving the value for property config.hardware.device: (vim.fault.NoPermission) {


• It is confirmed that the system SPS user is part of the 'Administrators' and 'ServiceProviderUsers' system groups using the following commands on the vCenter. The SPS user will have the naming convention of 'sps-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'. Grep for 'sps' when searching the output from the commands below: 
  
  /usr/lib/vmware-vmafd/bin/dir-cli group list --name ServiceProviderUsers
/usr/lib/vmware-vmafd/bin/dir-cli group list --name Administrators

Environment

VMware vCenter Server

Cause

• Even though the SPS user is part of the Administrator group, the 'VSPHERE.LOCAL\Administrators' group has no inventory permission set. You will see a log message similar to the following in the vCenter /var/log/vmware/vpxd-svcs/vpxd-svcs.log:
  
  [dataservice-4 [] WARN  com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterRiseImpl  opId=sps-DTCPoller-971626-324] User VSPHERE.LOCAL\sps-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX does not have privileges [System.Read] on object urn%3Avmomi%3AFolder%3Agroup-d1%3A09bc2a09-a42f-41db-9085-9d69cf9bf224
  
  
• The permission is missing due to manual changes made on the 'Administrators' group. When the change was made the propagate to children flag was accidentally set to to false. A log message similar to the following in the /var/log/vmware/vpxd-svcs/authz-event.log will confirm this:
  
  [tomcat-exec-262 [] INFO  AuthorizationService.AuditLog  opId=] Action performed by principal(name=<DOMAIN>\<USER-ID>,isGroup=false):Added access control [ Principal=Name=VSPHERE.LOCAL\Administrators,isGroup=true,roles=[-1],propagating=false ] to document urn:acl:global:permissions
  
  
• The above log confirms that a user made a change on the 'Administrators' group with propagating set to false. Since the 'Administrators' group is NOT propagating and the sps-* service account is part of that group, the account then loses permissions.

Resolution

• The solution is to ensure that the 'VSPHERE.LOCAL\Administrators' group permission has the propagates to children flag set to true like the screenshot below:
  
  
• To avoid the issue happening again, do not do any manual updates of the permissions in that system group, since it is vital for the operation of the vCenter.

Additional Information

• The issue might look similar to the KB313057, but that can be eliminated by confirming that the 'vmware-sps' service is running on the vCenter.