Troubleshooting Storage I/O Control

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on determining if SIOC has been correctly configured and provides steps to enable logging.

Symptoms:

• Storage I/O Control (SIOC) is not performing as expected.
• I/O from virtual machines does not get prioritized under congestion circumstances, or it gets prioritized when there is no real congestion.
• SIOC rules are intermittently applied to virtual machines on the same host.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Resolution

Storage I/O Control (SIOC) is used to control the I/O usage of a virtual machine and to gradually enforce the predefined I/O share levels. SIOC is supported on Fibre Channel and iSCSI connected storage in ESX/ESXi 6.X, 7.X and 8.X. Starting  ESXi 5.X the support for NFS with SIOC was also added.
For more information, see the Managing Storage I/O Resources section in the vSphere 7.0 Resource Management Guide or vSphere 8.0 Resource Management Guide.

Notes:

• Before using SIOC on datastores that are backed by arrays with automated storage tiering capabilities, check the VMware Storage/SAN Compatibility Guide to ensure that your automated tiered storage array is certified to be compatible with SIOC.
• Before enabling SIOC, ensure that datastores are managed by a single vCenter Server.
• Storage I/O Control does not support datastores with multiple extents. ( Cannot enable Storage IO Control on the datastore with extents )

Note: Storage I/O Control (SIOC) requires Enterprise Plus licensing. Without this license, the option to enable SIOC is grayed out.

Enabling Storage I/O Control

To enable SIOC:

1. Select a datastore in the vSphere Client inventory and click the Configuration tab.
2. Click Properties.
3. Under Storage I/O Control, select Enabled.
4. Click Close.

Note: This setting is specific to the datastore and not to the host.

If you experience problems with SIOC or if the number of hosts connected to the datastore has changed since enabling SIOC:

1. Disable SIOC and save the changes by clicking OK.
2. Enable SIOC and save the changes.

Determining if the threshold value has been modified

To determine if the threshold value has been modified:

1. Select a datastore in the vSphere Client inventory and click the Configuration tab.
2. Click Properties.
3. Under Storage I/O Control, click Advanced.
4. Check if the value is 30ms. If it is not 30, reset it to the default value of 30.

 

Ensuring virtual machines have disk shares assigned according to their importance

By default, all virtual machines have the same number of shares and IOPS limit. IOPS are the number of I/O operations per second. By default, IOPS are unlimited. If these defaults are not changed, then I/O control does not prioritize virtual machines.

To see the shares of all the virtual machines on the cluster, choose the cluster, click Resource Allocation, then click Storage.

To change the vDisk shares and limit:

1. Choose a virtual machine in the vSphere Client inventory.
2. Click the Summary tab and click Edit Settings.
3. Click the Resources tab and click Disk.
    
   • Choose a virtual hard disk from the list and click the Share column to select the relative amount of shares to allocate to the virtual machine (Low, Normal, or High).
   • You can also click Custom and enter a user-defined share value.
      
4. Click the Limit - IOPS column and enter the upper limit of storage resources to allocate to the virtual machine.
5. Click OK.

Note: The above process is used to set the resource consumption limits of individual vdisks in a virtual machine even when SIOC is not enabled. These settings are specific to the individual guest, and not the host, although they are used by SIOC.

Enabling and disabling SIOC logging on the host 

These logs are used for troubleshooting purposes. If you are sending the support logs to VMware, enable SIOC logging before collecting the logs. VMware recommends that you disable logging after the troubleshooting activity is complete.

To enable logging:

1. Click Host > Configuration.
2. In the left pane, click Software and then click Advanced Settings.
3. In the Misc section, select the Misc.SIOControlLogLevel parameter.
4. Set the value to 7 for complete logging.
   
   For example:
   
   Min value: 0 (no logging)
   Max value: 7
   
   After changing the log level, you see these changes logged in the/var/log/vmkernel logs:
   
   May 27 07:41:10 wdc-tse-h53 vmkernel: 76:14:16:03.334 cpu7:4110)Config: 297: "SIOControlLoglevel" = 1, Old Value: 0, (Status: 0x0)
    
5. Perform the action that is failing or repeat the procedure to replicate the observed issue.

To disable logging:

1. Click Host > Configuration.
2. In the left pane, click Software and then click Advanced Settings.
3. Set the Misc.SIOControlLogLevel value to 0.
4. After changing the log level, you see these changes logged in the /var/log/vmkernel logs.
   
   May 27 07:41:31 wdc-tse-h53 vmkernel: 76:14:16:24.521 cpu4:4111)Config: 297: "SIOControlLoglevel" = 0, Old Value: 1, (Status: 0x0)

Note: SIOC log files are saved in /var/log/messages. In ESXi 5.0 or later versions, the SIOC log files are saved in /var/log/storagerm.log.

Note: Setting the value for log level to "0" disabling debug logging but keeps Info. logging.

Run the below command to stop and start SIOC on ESX;

/etc/init.d/vmware-late {start|stop|status|restart}

Run this command to stop and start SIOC on ESXi:

/etc/init.d/storageRM {start|stop|status|restart}

Additional Information

For more information, see the Managing Storage I/O Resources section in the vSphere 7.0 Resource Management Guide or vSphere 8.0 Resource Management Guide.