Troubleshooting Zero I/O Throughput in ESXi During VM Workload Testing

Products

• VMware vSphere ESXi

Issue/Introduction

During a workload test on two virtual machines (VMs) in an ESXi environment, the I/O throughput for both VMs was reported as 0 bytes.
This article provides insights into the issue, offers troubleshooting steps, and suggests recommendations for future I/O performance testing.

VM Configuration Overview

The two virtual machines involved in the testing are configured as follows:

• VM 1: testvm001.example.com
• Configuration File Path:
  /vmfs/volumes/68f6ddXX-XXXXXXXX-XXXX-XXXXXX /testvm001.example.com/testvm001.example.com.vmx
• Virtual Disk:
  scsi0:0.fileName = "testvm001.example.com.vmdk"
• VM 2: testvm002.example.com
• Configuration File Path:
  /vmfs/volumes/68f6ddXX-XXXXXXXX-XXXX-XXXXXX/testvm002.example.com/testvm002.example.com.vmx
• Virtual Disk:
  scsi0:0.fileName = "testvm002.example.com.vmdk"

Both VMs have a single virtual disk (VMDK) attached, and their storage configuration is relatively simple.

Datastore and Device Information

The following datastore and device details were retrieved from the ESXi logs, using the relevant datastore UUIDs for both VMs:

• VM 1 (testvm001.example.com):
• VMFS Datastore: Datastore-01
• Datastore UUID: 68f6ddXX-XXXXXXXX-XXXX-XXXXXX
• NAA ID: naa.XXXXXXXXXXXX
• VM 2 (testvm002.example.com):
• VMFS Datastore: Datastore-02
• Datastore UUID: 68f6ddXX-XXXXXXXX-XXXX-XXXXXX
• NAA ID: naa.XXXXXXXXXXXX

The datastores, UUIDs, and NAA IDs have been anonymized to protect sensitive information.

 

Environment

vSphere ESXi 8
vSphere ESX 9

Cause

The observed issue of 0-byte I/O throughput is most likely due to the absence of I/O requests from the virtual machines during the workload test.
Since the ESXi logs show no errors related to I/O or disk activity, it appears the VMs simply did not generate any I/O activity at the time of testing, leading to the lack of recorded I/O throughput.

An analysis of the ESXi host logs from boot-up revealed the following key observations:

1. No Errors Related to Virtual Disks:
• The ESXi logs did not show any errors related to the virtual disks of these two VMs or their corresponding backend storage.
2. No VSCSI Reset Events:
• There were no VSCSI reset events in the ESXi logs, indicating stable communication between the virtual machines and the datastore.
3. Absence of I/O Errors:
• The logs did not report any I/O timeouts, I/O resets, or I/O drops. These types of errors often indicate issues with disk I/O but were absent during this period.
4. No Anomalies for NAA IDs:
• The logs for both NAA IDs and their corresponding paths did not show significant anomalies, such as performance degradation or hardware failures (e.g. 0x6 0x29, H:0x1).  

Resolution

To ensure accurate I/O performance testing for these virtual machines, follow these guidelines:

1. Ensure Load Generation Inside VMs:
• Use performance testing tools (e.g., fio, vdbench) inside the virtual machines to simulate an appropriate load and generate I/O requests. This will ensure the ESXi host records the corresponding I/O throughput.
2. Gradual Testing:
• Perform stress testing gradually by increasing the load in steps to observe system behavior under different levels of load and avoid overwhelming the system.
3. Monitor VM and ESXi Host Metrics:
• Continuously monitor the disk throughput, CPU, and memory usage on both the VMs and ESXi host during testing to identify any bottlenecks or resource limitations.
4. Consult Official Documentation:
• For detailed guidance on testing I/O performance, refer to the following Knowledge Base article:
  Testing Virtual Machine Storage I/O Performance for ESXi

By following these recommendations, you can ensure a more accurate assessment of virtual machine storage I/O performance and diagnose any issues that may arise.