vCenter Server experiences temporary performance degradation, 0% stuck tasks, and slow vMotion following upgrade or restart due to Horizon API flooding
search cancel

vCenter Server experiences temporary performance degradation, 0% stuck tasks, and slow vMotion following upgrade or restart due to Horizon API flooding

book

Article ID: 453646

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

After upgrading or restarting vCenter Server (e.g., updating from 8.0U3i to 8.0U3k), administrators may observe the following issues:

  • Task Queue Backlog: The Recent Tasks pane in vSphere Client is flooded with high volumes of Configure virtual disk digest and Reconfigure virtual machine tasks stuck at 0%.

  • vMotion Performance Drop: Initiating vMotion on VMs (including the vCenter Server Appliance itself) is extremely slow or appears unresponsive (taking 30–40+ minutes).

  • Alarm Fluctuation: Virtual machines trigger intermittent "CPU usage" and "Memory usage" status changes to "Gray".

  • Task Completion Recovery: Stuck tasks begin to complete normally, and vMotion speed restores, after the vCenter Server VM is successfully migrated to another ESXi host.

/var/log/vmware/vpxd/vpxd.log on the vCenter Server reveals excessive warnings indicating concurrent blocking calls from the Horizon Connection Server IP:
warning vpxd[3790863] [Originator@6876 sub=PropertyCollector opID=34d44e3a] Session ####-####-####-####-#### has issued 25 concurrent blocking calls to the property collector. This is almost certainly an error in the logic of the client application.

info vpxd[10072] [Originator@6876 sub=vpxLro opID=####] [VpxLRO] -- BEGIN task-3372 -- CbrcManager -- vim.CbrcManager.configureDigest

info vpxd[3153246] [Originator@6876 sub=VmProv opID=####-01] Applying ConfigSpec ... extraConfig = (key = "sched.mem.pshare.salt", value = "...")

The session ID from the above messages is found in vpxd-profiler log with an IP and Username belonging to the Horizon_Connection_Server:

$ grep "####-####-####-####-####" var/log/vmware/vpxd/vpxd-profiler-*.log
/SessionStats/SessionPool/Session/Id='####-####-####-####-####'//Username='VSPHERE.LOCAL\Administrator'/ClientIP='Horizon_Connection_Server_IP'

Environment

VMware vCenter Server 8.x

Cause

When vCenter restarts after an upgrade, Horizon Connection Server automatically re-syncs all VDI desktops. It sends a massive burst of simultaneous API requests to update storage cache settings (CbrcManager.configureDigest) and memory configurations (sched.mem.pshare.salt).

This API storm overloads vCenter's PropertyCollector and exhausts the vpxd service thread pool. As a result, tasks freeze at 0%, management responsiveness drops, and vMotion signaling slows down. Migrating the vCenter VM forces socket connections to reset, clearing the queue lockup.

Resolution

Preventative Steps (Before Upgrading or Restarting vCenter):

  1. Log in to the Horizon Console.

  2. Navigate to Desktops / Automated Desktop Pools.

  3. Temporarily Disable Provisioning for all active pools.

  4. Proceed with the vCenter Server upgrade or maintenance.

  5. Once vCenter Server is fully operational, Enable Provisioning in Horizon Console.

Remediation (If the Issue Currently Exists):

  1. Temporarily stop the VMware Horizon View Connection Server service on the Horizon Connection Server to halt the incoming API storm.

  2. Allow 5–10 minutes for vpxd thread queues to process and clear stuck tasks.

  3. Once task queues normalize, restart the Horizon Connection Server service.

Powered by Wolken Software