Poor vCenter Server performance after patching from 7.0.2.00500 to Version: 7.0.3.00600.

Products

• VMware vCenter Server

Issue/Introduction

After patching vCenter to version 7.0.3.00600 you experience the following symptoms:

• Edit settings takes a considerably long time.
• The UI freezes and navigating around the inventory is very slow.
• The dataservice.log shows slow execution whereas the vpxd log shows instant completion:
  
  • In the dataservice.log:
    
    [2022-08-05T18:42:19.251Z] [WARN ] http-nio-5090-exec-100       70001522 100009 200004 ProfilingLogService [queryId: q-1968:h5ui-getProperties:urn:vmomi:VirtualMachine:vm-362290:########-####-####-####-########166b:1206883735] The Data Service detected slow execution:
The query batch execution took too long: 120002 milliseconds.
    
    The vpxd.log shows the query (q-1968) completed at same second.
      
    2022-08-05T18:40:19.255Z info vpxd[09376] [Originator@6876 sub=vpxLro opID=q-1968:h5ui-getProperties:urn:vmomi:VirtualMachine:vm-362290:########-####-####-####-########166b:1206883735:01-61] [VpxLRO] -- BEG
IN lro-62629 -- ResourceModel -- vim.dp.ResourceModel.queryBatch -- ########-####-####-####-########be5c(########-####-####-####-########dfdf)
2022-08-05T18:40:19.255Z info vpxd[09376] [Originator@6876 sub=vpxLro opID=q-1968:h5ui-getProperties:urn:vmomi:VirtualMachine:vm-362290:########-####-####-####-########166b:1206883735:01-61] [VpxLRO] -- FINISH lro-62629

• There are a large number of http2 requests:
  
  less envoy-access-*.log | grep -c "external-vecs/http2"
47700

Environment

VMware vCenter Server 7.0.3
VMware vCenter Server 7.0.2

Resolution

This is resolved in vCenter Server 7.0 Update 3i (build number 20845200).

Workaround:

1. Take a snapshot of vCenter Server. 
2. Make a backup of /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd-rhttpproxy-endpoint.conf
3. Remove "http2:" from the following line:
   
   from
   /sdk http2:local 8085 redirect allow
   to
   /sdk local 8085 redirect allow
   
   
4. Restart httpproxy with the below command:
   
   service-control --restart rhttpproxy 


5. Restart the envoy front proxy with the below command:
   
   service-control --restart envoy

Additional Information