VMs have high latency and may freeze due to repeated D:0x28 (TASK_SET_FULL)
search cancel

VMs have high latency and may freeze due to repeated D:0x28 (TASK_SET_FULL)

book

Article ID: 423427

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • VMs experience high read and write latency.

  • VMs may freeze, fail, or undergo unexpected reboots/resets.

  • ESXi hosts may become unresponsive.

  • Issues typically occur during periods of high I/O load or scheduled tasks (e.g., backups).

  • Note: This often occurs in environments where high-I/O workloads (such as Oracle Database servers) share the same physical storage array ports or fabric with ESXi hosts.

Environment

VMware vSphere ESXi (all versions)

Cause

The storage device is overloaded and failing I/Os with SCSI status code D:0x28 (TASK_SET_FULL). This indicates the storage target cannot accept further commands because its internal resources such as command queues, cache buffers, or configured Quality of Service (QoS) limits are saturated.

Cause Verification:

Review /var/log/vmkernel.log for the following patterns

vmkwarning: cpu1:2098325)WARNING: ScsiDeviceIO: 1779: Device naa.###################### performance has deteriorated. I/O latency increased from average value of 1796 microseconds to 36501 microseconds.
…
vmkernel: cpu79:2098326)ScsiDeviceIO: 4619: Cmd(0x45da1181e1c0) 0x28, CmdSN 0x8000006a from world 7998099 to dev "naa.######################" failed H:0x0 D:0x28 P:0x0 

Resolution

If the issue occurs frequently or during typical workloads, it may indicate a storage sizing or fabric configuration issue.

  1. Fabric Segregation: Ensure that high-I/O non-ESXi workloads (e.g., Oracle DB) are not connected to the same storage array ports as the ESXi hosts to prevent port-level buffer exhaustion.

  2. Workload Redistribution: Redistribute intensive virtual machines across different datastores/LUNs to balance the load and avoid exceeding QoS limits on individual devices.

  3. Stagger Tasks: Stagger high-intensity operations (backups, antivirus scans, or heavy batch jobs) to reduce peak IOPS and throughput demand.

  4. Enable Adaptive Queuing: Configure ESXi to throttle I/O automatically when TASK_SET_FULL is detected. Refer to Controlling LUN queue depth throttling in VMware ESXi (311232).

  5. Storage Vendor Engagement: If host-side changes do not resolve the issue, engage your storage vendor (e.g., Infinidat, Dell, Pure) to analyze array-side CPU utilization, increase LUN queue depths, or adjust restrictive QoS limits.

Additional Information

To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.

Powered by Wolken Software