VCSA Performance issue or high CPU usage after upgrading from 7.0u2 to 7.0u3

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• High CPU seen on performance charts for the VCSA soon after upgrade/patching 
• Performance of VCSA is generally slow. The slowness may be isolated so some areas such as reviewing performance statistics for VMs or hosts.
• The vpostgres database logs show long duration time for some statements.
• These logs can be found on /var/log/vmware/vpostgres/
• Entries like the ones below can be found in the postgresql-01.log (you can replace 01 with current date, e.g on the 19th of the month look at postgresql-19.log)

Log Snippets:

LOG: duration: xxxxx ms statement: BEGIN;declare "SQL_CUR0x7f48d2fa0750" cursor with hold for SELECT sc.stat_id, d.device_name FROM vpx_stat_counter sc, vpx_device d WHERE sc.entity_id = 12535 and coalesce(sc.device_id,1) = coalesce(d.device_id,1) GROUP BY sc.stat_id, d.device_name ORDER BY sc.stat_id, d.device_name ;fetch 1024 in "SQL_CUR0x7f48d2fa0750"

LOG: duration: xxxxx ms statement: BEGIN;declare "SQL_CUR0x7ffa7dc92bf0" cursor with hold for SELECT EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME, USERNAME, CATEGORY, VM_ID, VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE, COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID, NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, STORAGEPOD_ID, STORAGEPOD_NAME, CHANGE_TAG_ID FROM VPXV_EVENT_ALL WHERE (EVENT_TYPE IN ('vim.event.VmMigratedEvent','vim.event.VmRelocatedEvent','vim.event.VmBeingRelocatedEvent','vim.event.VmSuspendedEvent')) AND EVENT_ID<ORDER BY EVENT_ID DESC LIMIT 10; ;fetch 1024 in "SQL_CUR0x7ffa7dc92bf0"

Cause

The high CPU usage is due to the vPostgreSQL service that is taking a very long time to query the tables. This could be confirmed using VIMTOP cmd run on the vCenter Appliance. 

Resolution

This issue has been fixed in vCenter Server 7.0.3 Update 3f, released on July 12th, 2022.
If you experience the issue, please ensure to patch your vCenter Server to this or a newer version.
 

Workaround:

The workaround for this issue is simple and can be run online. We suggest that every customer who has upgraded to VCSA 7.0u3 run this even if they do not currently experience an issue.

To implement the fix, you need to connect to the VCSA via ssh and login to the postgres database and run vacuum analyze

# /opt/vmware/vpostgres/current/bin/psql -U postgres VCDB -c "vacuum analyze;"