Storage I/O Control (SIOC) causing vCenter slowdown and database growth

Products

• VMware vCenter Server

Issue/Introduction

There are several common scenarios which can cause excessive growth of the vCenter Server database. One common cause is due to an issue with SIOC.

Symptoms:

• High disk usage of the vCenter Server database. Specifically the /storage/seat partition.
• Performance degradation in vCenter Server.

Environment

VMware vCenter Server 6.5.x

Cause

SIOC is using up CPU threads and flooding the database with SIOC data

Resolution

• In /var/log/vmware/vpxd.log you'll see lots of

2020-07-29T16:32:36.997+02:00 info vpxd[7F2B7952F700] [Originator@6876 sub=MoDatastore opID=HB-host-49367@140116-2729e4bc] [CheckForThresholdViolationInt] Datastore EPP_SHARED_PLATINUM10K_002 utilization(0.901093) > threshold(0.8); scheduling SDRS

2020-07-29T16:32:36.997+02:00 info vpxd[7F2B7952F700] [Originator@6876 sub=MoDatastore opID=HB-host-49367@140116-2729e4bc] [CheckForThresholdViolationInt] Datastore EPP_SHARED_PLATINUM10K_002 utilization(0.901093) > threshold(0.8); scheduling SDRS

• You can count these entries. In some instanes of this issue this count is 30796

grep -i "scheduling SDRS" vpxd.log * |wc -l

• We can also verify that this issue is being hit from the database. Run this command in the vCenter Server Appliance database:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres 

select event_type, count(event_type) from vpxv_event_all group by event_type order by count(event_type) desc limit 10;

 

if the event type vim.event.DatastoreIORMReconfiguredEvent is top of the list then you are hitting this know issue.                                                                              

Workaround:
Disable SIOC on your clusters

Upgrade to a 6.5P04, 6.7 or 7.0

Additional Information

Impact/Risks:
It will slow down your vCenter and eventually fill one of your disks causing vCenter to crash.