Poor vCenter Server performance on vCenter UI while searching for objects and expanding the inventory folders.

Products

• VMware vCenter Server

Issue/Introduction

vCenter experience the following symptoms:

- Edit settings takes a considerably long time.

- The UI freezes and navigating around the inventory is very slow.

- The dataservice.log shows slow execution whereas the vpxd log shows instant completion:

 

In the var/log/vmware/vpshere-ui/logs/dataservice.log

YYYY-MM-DDTHH:MM.SSSZ [WARN ] http-nio-5090-exec-100       70001522 100009 200004 ProfilingLogService [queryId: q-1968:h5ui-getProperties:urn:vmomi:VirtualMachine:vm-######:########-####-####-####-########166b:1206883735] The Data Service detected slow execution:
The query batch execution took too long: 120002 milliseconds.

The vpxd.log shows the query (q-1968) completed at same second.
  
YYYY-MM-DDTHH:MM.SSSZ info vpxd[09376] [Originator@6876 sub=vpxLro opID=q-1968:h5ui-getProperties:urn:vmomi:VirtualMachine:vm-######:########-####-####-####-########166b:1206883735:01-61] [VpxLRO] -- BEG
IN lro-62629 -- ResourceModel -- vim.dp.ResourceModel.queryBatch -- ########-####-####-####-########be5c(########-####-####-####-########dfdf)
YYYY-MM-DDTHH:MM.SSSZ info vpxd[09376] [Originator@6876 sub=vpxLro opID=q-1968:h5ui-getProperties:urn:vmomi:VirtualMachine:vm-######:########-####-####-####-########166b:1206883735:01-61] [VpxLRO] -- FINISH lro-62629


There are a large number of http2 requests:

less envoy-access-*.log | grep -c "external-vecs/http2"
47700

Environment

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

Resolution

This issue is resolved in vCenter Server 7.0 Update 3i (build 20845200). For environments running version 8.0, the following steps may be applied as a workaround:

Workaround:

1. Take a snapshot of vCenter Server. 
2. Make a backup of cp /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd-rhttpproxy-endpoint.conf /storage/core
3. Remove "http2:" from the following line:
   
   from
   /sdk http2:local 8085 redirect allow
   to
   /sdk local 8085 redirect allow
   
   
4. Restart httpproxy with the below command:
   
   service-control --restart rhttpproxy 


5. Restart the envoy front proxy with the below command:
   
   service-control --restart envoy

Additional Information