StorageRM excessive logging on ESXi 6.7 P01 and 7.0

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on disabling SIOC logging.

Symptoms:

• You use vSphere 6.7 or 7.0 environment
• Storage I/O Control is enabled and in use
• Even after patching ESXi's to 6.7 P01 or 7.0 StorageRM still has excessive logging 
• StorageRM logs generates more than 50 000+ lines of messages within couple of minutes
• This has a huge impact on external syslog remote host
• Following information on disabling SIOC logging from KB Troubleshooting Storage I/O Control (1022091) doesn't reduce the excessive logging because at level 0, it will still be printing the INFO level logs

Environment

VMware vSphere 7.0.x
VMware vSphere ESXi 6.7

Cause

• These log entries are normal and occurring due to bigger Clusters reaching up to maximum of 64 ESXi hosts
• Whenever storageRM updates the cluster parameters, we will have one entry per host and that is why we see a lot of logs being printed.
• This function goes through the stats of all the hosts accessing that device and computes cluster wide aggregation of various stats such as iocount, latency, outstanding IOs and window sizes.

Resolution

The issue is addressed in VMware ESXi 6.7 P02 and 7.0 U1. The improvements have been added to the StorageRM logging.

Workaround:
You can use following to temporarily disable SIOC logging:

in file: /etc/vmsyslog.conf.d/storageRM.conf
change:
fclass = FileLoggerSyslog
to
fclass = FileLoggerNull

Please make the same change to /etc/vmsyslog.conf.d/sdrsInjector.conf

Save the files and then run  
esxcli system syslog reload

This will stop storageRM and sdrsInjector logging.

NOTE: The above will only stop logging to a log file stated, however it will still send log lines to external remote syslog server. In order to completely stop excessive logging use below:

a. Edit /etc/vmware/logfilters and add the following lines:
5 | storageRM | .*
5 | sdrsInjector | .*

b. Add the line "enable_logfilters = true" to /etc/vmsyslog.conf

c. Run the command: "esxcli system syslog reload" for the configuration to take effect.

Logging should automatically stop after 5 log statements for both storageRM and sdrsInjector.

Additional Information

Impact/Risks:
There is no impact of disabling SIOC logging. SIOC will continue to work fine and will dynamically adjust queue depths according to the data store latencies and IO limits.