vCenter Server 7.x and 8.x becomes unresponsive during high-frequency role updates

Products

VMware vCenter Server

Issue/Introduction

vCenter Server becomes intermittently unresponsive to Web Client and API requests.
The issue has been observed after upgrading to vCenter Server 8.0 Update 3g (build 24853646).
Automation scripts using PowerCLI or APIs to perform multiple role updates (updateRole()) encounter timeouts or hangs.
Checking service status shows all services "Running," but the vpxd-svcs (Authorization) service is locked.
In vpxd-svcs.log, you may observe repeated authorization service timeouts or race condition indicators.

Messages in /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log (virgo log) like the following:

[INFO ] data-service-pool-######### c.v.license.client.cis.ls.authz.impl.VpxAuthorizationManagerImpl VC's managed object reference was not found. Could be due to having 'No Access' privileges on VC level. com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: ManagedObjectReference: type = Folder, value = group-d1, serverGuid = ######## refers to a managed object that no longer exists or has never existed.
        at com.vmware.vise.vim.data.util.QueryDataUtil.newObjectNotFoundFault(QueryDataUtil.java:351)
        at com.vmware.vise.vim.data.util.QueryDataUtil.throwIfObjectNotFound(QueryDataUtil.java:198)
        at com.vmware.vise.vim.data.util.QueryDataUtil.getProperties(QueryDataUtil.java:90)
        at com.vmware.vsphere.client.security.impl.VcAuthorizationServiceImpl.getPrivilegesInternal(VcAuthorizationServiceImpl.java:112)
........

/var/log/vmware/vsphere-ui/logs/vsphere_UI.log (vsphere UI log) has messages like:

[ERROR] live-updates-executor-195 70071819 116006 200189 DataServiceImpl [queryId: vb-###] Error occurred while executing query:
The query sent to the Data Service is:

QuerySpec
QueryName: auto-name-####
ResourceSpec
  Constraint: ObjectIdentityConstraint
   TargetType: null
   Target: ManagedObjectReference: type = Folder, value = group-d1, serverGuid = ############
  PropertySpec[1]
   ProviderType: Folder
   Relationship: null
   Properties[2]
    PropertyName: parent
    PropertyName: parentName
ResultSpec:
  Offset: null
  MaxResultCount: null
  OrderingCriteria
   OrderPropertySpec[1]
    ProviderType: Object
    Relationship: null
    SortType: ASCENDING
    OrderingProperties[1]
     PropertyName: id
Options: null
com.vmware.vise.data.query.internal.DataServiceException: The query execution timed out because of a back-end data adapter 'com.vmware.vise.data.adapters.core.DataServiceCoreAdapter' which took more than 120 seconds.
        at com.vmware.vise.data.query.impl.DataAdapterUtil.processDataAdapterTaskException(DataAdapterUtil.java:159)
        at com.vmware.vise.data.query.impl.DataAdapterUtil.executeAdapterTasks(DataAdapterUtil.java:125)
        at com.vmware.vise.data.query.impl.DataAdapterRetriever.runDataTasks(DataAdapterRetriever.java:374)
        at com.vmware.vise.data.query.impl.DataAdapterRetriever.invokeDataAdapters(DataAdapterRetriever.java:325)
........
Caused by: java.util.concurrent.TimeoutException: null
        at com.vmware.vise.util.concurrent.ExecutorUtil.executeTasks(ExecutorUtil.java:744)
        at com.vmware.vise.data.query.impl.ProfiledTaskExecutor.executeTasks(ProfiledTaskExecutor.java:70)
        at com.vmware.vise.data.query.impl.DataAdapterUtil.executeAdapterTasks(DataAdapterUtil.java:112)
        ... 27 common frames omitted

Environment

vCenter 7.x and 8.x

Cause

This issue has been caused by a race condition/deadlock within the vpxd-svcs authorization service. High-frequency concurrent role updates can cause the internal authorization manager to lock up. This is specifically seen in environments where scripts perform multiple role modifications in rapid succession.

Resolution

There is currently no permanent fix in vCenter 8.0 Update 3. A permanent resolution would require a significant redesign of the authorization system, which is being evaluated for future major releases.

Additional Information

Workaround

To restore vCenter accessibility without restarting all services, you can perform a targeted restart of the vpxd-svcs component.

Method 1: PowerCLI (Recommended)

Use the following command to specifically target and restart the affected authorization service:

Get-VIApplianceService 'vpxd-svcs' | Restart-VIApplianceService

Method 2: VCSA Command Line (SSH)

If PowerCLI is unavailable, you can restart the service via the vCenter Server Appliance shell:

service-control --restart vmware-vpxd-svcs

Method 3: Script Optimization

To reduce the occurrence of this deadlock:

Reduce Frequency: Consolidate multiple updateRole or addPrivilege calls into a single batch where possible.
Add Latency: Introduce a brief delay (e.g., Start-Sleep) between role modification calls in your automation scripts to reduce the likelihood of a race condition.