Guest OS Isolated Virtual SCSI Reset in VMware vSphere

Products

• VMware vSphere ESXi

Issue/Introduction

This article addresses instances where a virtual machine's internal monitoring agents or OS logs report retried disk I/O operations or resets, while the physical storage layer remains healthy.

• Primary Product: VMware vSphere ESXi
• Component: Virtual Machine Management / Virtual Storage
• Symptoms:
  • Guest OS logs report messages such as:The IO operation at logical block address 0x###### for Disk 0 (PDO name: \Device\########) was retried
  • Internal monitoring agents (e.g., AltosAgent) flag transient disk timeouts.
  • No corresponding hardware or storage fabric errors are found in the ESXi host event logs.

Environment

ESXi 8.0

Cause

The issue is caused by a virtual SCSI (vSCSI) reset command originating explicitly from the Guest OS layer (Virtual Machine Monitor - VMM).

This occurs as a routine self-healing mechanism within the Guest OS when a specific application block request experiences a transient delay that exceeds the default OS disk timeout parameter. This is typically triggered during brief, intense bursts of local I/O utilization within the VM. Because the reset is initiated by the guest and not the hypervisor, the ESXi vmkernel logs will show the reset source as vmm0:

VSCSI: 3807: handle <handle_id>(GID:<gid>)(vscsi0:0):Reset [Retries: 0/0] from (vmm0:<vm_name>)

Verification of the ESXi infrastructure during these events typically shows:

• No concurrent SAN fabric or physical device link failures.
• Storage latency remains within nominal thresholds.
• No SCSI sense code errors or out-of-space conditions on the datastore.

Resolution

Since these events are typically singular and transient, they recover instantly without ongoing system degradation. In such cases, no configuration changes are required on the ESXi host or storage array.

If the warnings occur systematically or frequently, perform the following troubleshooting steps:

1. Verify Source of Reset: Check the ESXi vmkernel.log to confirm the reset is coming from the VM monitor (vmm0) and not the storage driver.
2. Evaluate I/O Load: Inspect the Guest OS application logs and task scheduler for I/O thread bottlenecks during the timestamp of the alert.
3. Adjust Disk Timeouts: If the application workload is known to produce high-intensity bursts, consider increasing the Guest OS disk timeout parameters (e.g., Disk.TimeOutValue in Windows or udev rules in Linux).
4. Monitor Queue Depth: Review internal application trace logs to evaluate if the disk queue depth requires optimization for the specific workload.