Unmanaged workload is detected on datastore running SIOC

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This alarm may appear in the vCenter vSphere Client:

Non-VI workload detected on the datastore

A warning message similar to one of these may also appear in the vCenter vSphere Client:

An external I/O activity is detected on datastore “datastore”, this is an unsupported configuration. Consult the Resource Management Guide or follow the Ask VMware Link for more information

An external I/O workload is detected on datastore “datastore”

 

This informational event alerts the user of a potential misconfiguration or I/O performance issue caused by a non-ESXi workload. It is triggered when Storage I/O Control (SIOC) detects that a workload that is not managed by SIOC is contributing to I/O congestion on a datastore that is managed by SIOC. (Congestion is defined as a datastore's response time being above the SIOC threshold.) Specific situations that can trigger this event include:

• The storage array is performing a system operation such as replication or RAID reconstruction.
• VMware Consolidated Backup or vStorage APIs for Data Protection are accessing a snapshot on the datastore for backup purposes.
• The storage media (spindles, SSD) on which this datastore is located is shared with volumes used by non-vSphere workloads
• The host is running in an unsupported configuration.

SIOC continues to work during these situations. This event can be ignored in many cases and you can disable the associated alarm once you have verified that none of the potential misconfigurations or serious performance issues are present in your environment. As explained in detail below, SIOC ensures that the ESXi workloads it manages are able to compete for I/O resources on equal footing with external workloads. This event notifies the user of what is happening, provides the user with the opportunity to better understand what is going on, and highlights a potential opportunity to correct or optimize the infrastructure configuration.

 

Note: Starting with vSphere 5.0, SIOC is supported with NFS storage. However, Raw Device Mapping (RDM) virtual disks are not supported. This includes both Physical Mode and Virtual Mode RDMs as well as RDMs used for Microsoft Cluster Server (MSCS). Datastores with multiple extents are also not supported with SIOC. This alarm could be triggered if these storage objects are configured.
 

This article helps you understand what is causing the event, and what, if any, action is required to address the cause.

Example

The array being used for vSphere is also being used for non-vSphere workloads. The non-vSphere workloads are accessing a storage volume that is on the same disk spindles as the affected datastore.

Impact

When SIOC detects that datastore response time has exceeded the threshold, it typically throttles the ESXi workloads accessing the datastore to ensure that the workloads with the highest shares get preference for I/O access to the datastore and lower I/O response time. However, such throttling is not appropriate when workloads not managed by SIOC are accessing the same storage media. Throttling in this case would result in the external workload getting more and more bandwidth, while the vSphere workloads get less and less. Therefore, SIOC detects the presence of such external workloads, and as long as they are present while the threshold is being exceeded, SIOC competes with the interfering workload by reducing its usual throttling activity.

SIOC automatically detects when the interference goes away and resumes its normal behavior. In this way, SIOC is able to operate correctly even in the presence of interference. The vCenter Server event is notifying the user that SIOC has noticed and handled the interference from external workloads.

Note: When an external workload is acting to drive the datastore response time above the SIOC threshold, the external workload might cause I/O performance issues for vSphere workloads. In most cases, SIOC can automatically and safely manage this situation. However, there may be an opportunity to improve performance by changing some aspects of your configuration. The next section provides guidance on this.

Environment

VMware vSphere ESXi 8.0
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 5.5
VMware ESXi 4.1.x Embedded
VMware ESX 4.1.x
VMware vSphere ESXi 5.1
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.0

Resolution

These unsupported configurations can result in the triggering of the alarms or the Warning message as seen above:

• One or more hosts accessing the datastore are not managed by vCenter Server.
• Not all of the hosts accessing the datastore are managed by the same vCenter Server.
• The storage media (spindles, SSD) where this datastore is located are shared with other datastores that are not SIOC enabled.
• Datastores in the configuration have multiple extents.

Ensure that you are not running the above configurations when using SIOC for Congestion Management:

 

Note: SIOC can be used even when External Workloads are using the same underlying storage array as vSphere. This could be detrimental to the I/O performance of the virtual machines. To function in presence of external workloads, SIOC needs to throttle virtual workloads; around 10% average if the external workload interference is continuous. If this is not acceptable, currently, disabling SIOC is the only solution. But, if the external workload is temporal, for example, for a few hours and there is sufficient capacity on the array, enabling SIOC acts as an insurance against IO performance problems that may happen due to mis-configurations in a shared storage virtual environment.

• Can you disable and successfully re-enable congestion management for the affected datastore?

• Are hosts that are not managed by this vCenter Server accessing the affected datastore?

If disabling and re-enabling congestion management for the affected datastore does not solve the problem, other hosts that are not managed by this vCenter Server might be accessing the datastore.

 

Verify that the datastore is shared across hosts that are managed by different vCenter Server systems or are not managed hosts. If so, perform one of these actions:

• • Add all the hosts that access the affected datastore to the same vCenter Server or
  • Migrate the datastore's workloads to hosts that are managed by the same vCenter Server.
     
• Do all datastores in the configuration that share the same physical storage media (spindles, SSD) have the same SIOC configuration?

  All datastores that share physical storage media must share the same SIOC configuration — all enabled or all disabled. In addition, if you have modified the default congestion threshold setting, all datastores that share storage media must have the same setting.

• Are any SIOC-enabled datastores in the configuration backed up by multiple extents?

  SIOC-enabled datastores must not be backed up by multiple extents.

If none of the above scenarios apply to your configuration and you have determined that you are running a supported configuration, but are still seeing this event, investigate possible I/O throttling by the storage array.

If an environment is known to have shared access to datastores or performance constraints, it may be preferable to disable the Alarm in vCenter Server. 

 

This flowchart describes the troubleshooting process for VMware Storage I/O Control:

 

To disable the Alarm:

1. Log in to vCenter Server using the vSphere Client.
2. Click the vCenter Server in the Inventory.
3. Click the Alarms tab.
4. Click the Definitions sub-tab. A list of defined alarms is displayed.
5. Right-click the Unmanaged workload is detected on the datastore alarm and click Edit Settings.
6. Deselect Enable this alarm.

"The /var/log/vobd.log file in the ESXi host may contain

< Detected a non vi workload on datastore > followed by the friendly name of the affected VMFS datastore."