Device latency with 'performance has deteriorated' messages in ESXi host logs

Products

• VMware vSphere ESXi

Issue/Introduction

• Devices are reporting high latency; virtual machines may become unresponsive during I/O operations.
• Virtual machines may appear to "freeze" or experience brief pauses. In severe cases, virtual disks may disconnect, or guest OS filesystems may be marked as read-only.
• Increased I/O latency as reflected in log messages has been observed.
• Storage latency alerts have been triggered at the vCenter level.
• High I/O wait is observed for VMs from the application level.
• VM clone tasks are getting queued.
• In TKGS environments, Supervisor pods (etcd, kube-apiserver) may restart due to latency.

Environment

VMware vSphere ESXi

Cause

The issue may occur when:

• The ratio of latency is significantly higher than previous logging.
• The latency ratio has doubled since the last log was written.

The device latency may increase due to:

• Changes made to the target.
• Disk or media failures.
• Overload conditions on the device.
• A failover event.

The numbers reported in the events are in microseconds and refer to the DAVG measurements in the esxtop storage screen.

Refer: Using esxtop to identify storage performance issues for ESXi

Latency is a measure of the round-trip time between the issuance of a SCSI command from the hypervisor, through the transport to the surface of the media, and the return. Therefore, the source of the delay could be anywhere in the fabric, the storage infrastructure, or anywhere along the storage path.

Performance Benchmarks:

• Non-flash technologies: The generally accepted threshold is about 10 milliseconds (10,000 microseconds).
• Flash-based storage: It is rare to see DAVG latency above 1-2 milliseconds; these events should be investigated if latency exceeds this range.

From Esxi, var/run/log/vmkernel.log there is an event for "performance has deteriorated"

[YYYY-MM-DDTHH:MM:SS] cpu51:2098041)WARNING: ScsiDeviceIO: 513: Device naa.########## performance has deteriorated. I/O latency increased from average value of 38762 microseconds to 776315 microseconds.
[YYYY-MM-DDTHH:MM:SS] cpu47:2098037)WARNING: ScsiDeviceIO: 1443: Device naa.######### performance has deteriorated. I/O latency increased from average value of 12017 microseconds to 254228 microseconds.
[YYYY-MM-DDTHH:MM:SS] cpu47:2098038)WARNING: ScsiDeviceIO: 1216: Device naa.######### performance has deteriorated. I/O latency increased from average value of 18057 microseconds to 534229 microseconds.

Resolution

1. High device latency:

   • If device latency is high consistently, check the storage performance. If failures are logged on the storage array side, contact the storage vendor for further assistance.
   • Check if these messages are generated during scheduled tasks (e.g., backups or replications), which can cause intermittent performance problems.
   • Driver Compatibility: Verify that your host is running supported drivers for your HBA hardware. Please refer to the Broadcom Compatibility Guide to ensure your configuration is supported for your specific ESXi version.

2. Overload conditions:

   • If the message is generated due to an overload condition, reduce the load on the affected storage device.

3. Latency duration:

   • Examine the vmkernel.log to determine the time elapsed between the "performance has deteriorated" log entry and the "performance has improved" log entry.
   • The two corresponding lines can usually be determined by matching the device ID (naa, eui, etc.) and confirming that the "latency increased" to value matches the "latency improved" from value.

Additional Information

Characterizing Observed Latency: Use the following framework to investigate:

• Magnitude: How high are the spikes in DAVG?
• Duration: How long does each spike last?
• Frequency: What pattern is exhibited by the date and time stamps?
• Scope: Are events on one datastore or multiple? One host or across clusters?

For more information, see Using esxtop to identify storage performance issues for ESXi.

Note: ESXi does not cause latency spikes; it merely reports them. While the root cause cannot be determined from the ESXi perspective alone, the data above guides an investigation into the external storage infrastructure.