Large amount of datastore.refresh tasks cause vCenter to become unresponsive when connecting to it via VMware Workstation

Products

• VMware Desktop Hypervisor
• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter inventory is slow to respond or non-responsive in the Web Client.
• A huge amount of Datastore.refresh tasks are seen in the /var/log/vmware/vpxd/vpxd.log
  • NOTE: The following command can be used to determine this:

grep 'BEGIN.*.Datastore.refresh' /var/log/vmware/vpxd/vpxd.log | wc -l
 

• There are users that log in to the vCenter Server environment via VMware Workstation.

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 7.0.x
VMware vCenter Server 6.7.x

Cause

This issue is caused by a race condition in VMware Workstation that continually attempts to retry a datastore refresh task on the vCenter until vCenter eventually exhausts the available pool of consistent LRO tasks. It is commonly seen when a user uses VMware Workstation to login to a vCenter Server while inventory updates are simultaneously occurring (i.e. datastore/host/vm removal or add operations).

Resolution

This issue has been fixed in VMware Workstation 16.2

Workaround:
To workaround this issue, the vpxd service can be restarted, or the offending VMware workstation user can disconnect from the vCenter session. Locating which user this is can be difficult if there are many people using this method to connect to vCenter however.

Additional Information

This issue can also be detected in the vpxd-profiler logs. By looking at the VpxLRO internal queued tasks metric. This number in addition to the running total should not exceed 5000.

/var/log/vmware/vpxd/vpxd-profiler.log
--> /SystemStats/ThreadPool/VpxLro/Internal/Queued/total 4618
--> /SystemStats/ThreadPool/VpxLro/Internal/Running/total 608