vSAN performance diagnostics reports: "The outstanding IOs for the benchmark might not be optimal to achieve the desired goal"

Products

• VMware vSAN
• VCF Operations

Issue/Introduction

This article explains the vSAN performance diagnostics issue alert. 

• "The outstanding IOs for the benchmark might not be optimal to achieve the desired goal"

This article will show why it might be showing up, and what are possible solutions to address the issue.

Symptoms:

In the vSAN performance diagnostics, you see this message:

• The outstanding IOs for the benchmark might not be optimal to achieve the desired goal

Environment

• VMware vSAN 6.x
• VMware vSAN 7.x
• VMware vSAN 8.x
• VMware vSAN 9.x
• VCF Operations 9.1

Cause

• This message means that the outstanding IOs queued by the benchmark are either too low or too high for the desired goal, which means that the benchmark workload is either insufficient or too much for the desired goal.
• The detailed performance graphs explain how this error is detected. The graphs may show the plot of outstanding IOs at the vSAN –VM consumption (that is, outstanding IOs across the vSAN cluster) and compare against an expected threshold. A screenshot is displayed below.

• The detailed graphs may also show the “Delayed IO Percentage” at the vSAN disk group layer or the “Number of IOPS delayed” at the VM layer and compare against the expected threshold. The detailed graphs may also compare the vSAN backend with disk groups and demonstrate that the vSAN backend latency is much higher than the disk group latency, thereby implying that there is high queuing at the vSAN DOM Component Manager layer due to high outstanding IOs issued by the benchmark.

 

Resolution

What are possible ways to address the issue?

If you encounter this exception, the remedy is to adjust the benchmark workload that you are running on the vSAN cluster according to the desired goal of performance evaluation. To achieve higher IOPS or higher throughput, it may be necessary to increase the outstanding IO. To achieve lower latency, it may be necessary to decrease the outstanding IO. More details follow:

• If your goal is maximum IOPS or maximum throughput, you must configure your benchmark to execute high outstanding IOs (ideally at least 2 per host in the cluster. As an example, for a 4 node cluster, the benchmark must drive a cumulative of 8 outstanding IOs in its workload, the higher, the better). Otherwise, the system does not have a sufficient number of parallel IOs to keep all the storage devices busy with IOs. For both of these goals, you must keep increasing the outstanding IOs in your benchmark, until you hit a point where the system runs into vSAN disk group congestion (See KB: vSAN performance diagnostics reports: "vSAN is experiencing congestion in one or more disk group(s)"), or the result of the benchmark (IOPS or Throughput) does not get any higher. If your benchmark does not allow you to specifically tune the outstanding IOs, you may increase the number of VMs on the cluster to increase the number of outstanding IOs.
• If your goal is minimum latency, you must configure your benchmark to execute low outstanding IOs (ideally 1 per host); otherwise, the IOs may queue at different layers, leading to higher latency. If your benchmark does not allow you to specifically tune the outstanding IOs, you may decrease the number of VMs on the cluster to decrease the number of outstanding IOs.

How do I configure this parameter in HCIBench?

 

Please set the parameter of Number of Threads Per Disk in the Vdbench Parameter File Generate Page of HCIBench.

• You must have at least two threads per disk for the goals of maximum IOPs and maximum throughput.
• You must have one thread per disk for the goal of minimum latency.

Alternatively, run the “Easy Run” mode of HCIBench.

 

*In vSAN 8.x there is a built in performance benchmark analyzer that can be used to get similar test and results.