SQL Datastore Latency Observed on iSCSI Storage After Aggregate Server Outage/Reboot

Products

• VMware vCenter Server 6.0
• VMware vCenter Server 8.0

Issue/Introduction

This article provides guidance on identifying and resolving SQL datastore latency observed on iSCSI storage following an aggregate server outage or reboot. Such outages can result in backlog issues, inconsistent data flow, and degraded SQL performance.

Environment

VMware vSphere/ESXi hosts

Microsoft SQL Server with multiple disks

iSCSI storage

Cause

Aggregate server outage/reboot, SQL datastores may show latency in iSCSI storage. This latency manifests as backlog issues and inconsistent data flow across the environment.

Resolution

Perform a storage rescan on the ESXi host after server outage/reboot.

Steps to Rescan Storage on an ESXi Host (vSphere Client)

Log in to the vSphere Client

Select the ESXi host

Navigate to Configure > Storage > Storage Adapters

Click Rescan All to detect new or changed storage devices and LUNs

CLI/API Command Snippets for Storage Rescan
ESXi Host CLI (SSH):

bash
Rescan all HBAs on the host
esxcli storage core adapter rescan --all

Rescan all VMFS volumes
esxcli storage filesystem rescan
vSphere PowerCLI:

powershell
Connect to vCenter
Connect-VIServer <vcenter-fqdn>

Rescan all HBAs on a specific host
Get-VMHost -Name <esxi-hostname> | Get-VMHostStorage -RescanAllHba

Rescan all datastores on a specific host
Get-VMHost -Name <esxi-hostname> | Get-VMHostStorage -RescanVmfs

 

Additional Information

Verification Commands
After performing a storage rescan, validate that latency has normalized and all paths are visible.

ESXi Host CLI (SSH):

bash
Check device paths and ensure all are active/available
esxcli storage core path list

Verify datastore visibility
esxcli storage filesystem list
Using esxtop (real-time performance monitoring):

SSH into the ESXi host

Run:

bash
esxtop
Press u to view disk device statistics

Monitor CMDS/s (commands per second), LAT/rd (read latency), and LAT/wr (write latency)

Normal values are typically < 20 ms for reads/writes in healthy environments

Press v to view VM disk statistics for per-VM latency

Optional: vscsiStats (per-VM I/O distribution):

bash
# List all VM world IDs
vscsiStats -l

# Collect stats for a specific VM world ID
vscsiStats -s -w <worldID>

# Display histogram of I/O latencies
vscsiStats -p all -w <worldID>
Solution Justification
After rescanning, esxtop was used to validate performance