Excessive queue depth and StorageRM log messages with SIOC enabled

Products

• VMware vSphere ESXi

Issue/Introduction

This KB is written to advise of this problem, and provide options to workaround and when a fix is available.

Symptoms:
vSphere 6.7, 7.0, or 8.0 is in use.
Storage IO Control (SIOC) is enabled.
vmkernel.log begins showing excessive amounts of messages similar to below (excessive being many thousand to millions per host where SIOC is enabled):

Dates, times, and other designators such as datastore will vary

<TIMESTAMP> cpu32:2099143)ScsiVsi: 2225: Can't set the maxPathQueueDepth value to more than device advertised maxPathQueueDepth 32 

<TIMESTAMP> storageRM[2099092]: Default max Qdepth for datastore <DATASTORE_NAME> is 64 

Environment

VMware vSphere 6.7.x
VMware vSphere ESXi 7.0

Resolution

This will be resolved in vSphere 7.0 P07/7.0 U3i and 8.0 U1.

Workaround:
Two options exist as listed below:
 

1. Leave SIOC disabled until the code fix is available. Once the fix is available work to upgrade hosts to the version with the fix before enabling again. This will prevent the log spew, however would leave VMs vulnerable to impact from noisy neighbors generating excessive unchecked IO.
2. Enable SIOC as best practice to avoid impact from excessive IO use from Virtual Machines during normal production operations. The excessive logging will continue until the code fix is released and all hosts upgraded. The log spew will have no impact on normal operations, but will fill logs quickly and could have impact to syslog servers if they cannot keep up with the additional logging.

Additional Information

Impact/Risks:
A discrepancy is seen between the advertised/actual maximum queue depth and a value attempting to be set by StorageRM. VSI fails to obtain the correct value from PSA when queried and returns a default value of 64 for queue depth, which results in the queue depth messages logged if the actual maximum is below 64 as the depth cannot be set above the actual maximum.