Database Virtual Machine Performance Degradation Due to Small Block Write Latency on vSAN RAID-5

Products

• VMware vSAN

Issue/Introduction

Database workloads running on virtual machines (VMs) experience slowness and I/O latency in the guest operating system.
The guest OS reports "wait for read I/O" errors, and database operations show reduced throughput (~1 MB/s).

Observed behavior includes:

• Reduced performance during small block write operations using oflag=dsync.

[root@pod~]# dd if=/dev/zero of=/gpfs/network/testfile 2025 bs=4096 count=1000 oflag=dsync
1000+0 records in
1000+0 records out
1096000 bytes (4.1 MB, 3.9 MiB) copied, 2.80022 s, 1.5 MB/s

• Throughput increases for larger block sizes (≥64 KB).

root@pod~]# dd if=/dev/zero of=/gpfs/network/testfile_2025 bs=64K count=1000 oflag-dsync
000+0 records in
1000+0 records out
55536000 bytes (66 MB, 62 MiB) copied, 8.45999 s, 7.7 MB/s
root@pod~]# dd if=/dev/zero of=/gpfs/network/testfile_2025 bs=1M count=1000 oflag-dsync
1000+0 records in
1000+0 records out
04#####000 bytes (1.0 GB, 1000 MiB) copied, 10.8796 s, 96.4 MB/s

• Backend vSAN health and performance metrics remain normal.
• The issue affects database VMs using thin-provisioned disks on RAID-5 (FTT=1) storage policy.

 

Environment

VMware vSAN 8.x

VMware vSAN 7.x

Cause

The performance degradation is primarily caused by:

• RAID-5 Parity Overhead
  • RAID-5 introduces parity computation that multiplies physical I/O operations for every logical write.
  • This effect is most pronounced for small, random, synchronous writes (such as database I/O).
• Non–4K-Aligned I/Os on Namespace Objects
  • vSAN reports non–4K-aligned I/Os when VM snapshots are retained for extended periods (typically >72 hours).
  • The .vmsd file within each VM’s namespace stores snapshot metadata such as hierarchy and configuration.
  • Frequent updates to this file occur when snapshots are created, consolidated, or deleted.
  • These updates are small (<4 KB) and not 4K-aligned, resulting in continuous namespace-level writes.

•  

The combination of RAID-5 parity overhead and long-retained snapshots contributes to the observed latency and slow database response.

Validation

dd performance testing within the guest OS confirmed low throughput for 4 KB synchronous writes (~1–1.5 MB/s) and higher performance for large block writes (>60 MB/s).

[root@pod~]# dd if=/dev/zero of=/gpfs/network/testfile 2025 bs=4096 count=1000 oflag=dsync
1000+0 records in
1000+0 records out
1096000 bytes (4.1 MB, 3.9 MiB) copied, 2.80022 s, 1.5 MB/s
root@pod~]# dd if=/dev/zero of=/gpfs/network/testfile_2025 bs=64K count=1000 oflag-dsync
000+0 records in
1000+0 records out
55536000 bytes (66 MB, 62 MiB) copied, 8.45999 s, 7.7 MB/s
root@pod~]# dd if=/dev/zero of=/gpfs/network/testfile_2025 bs=1M count=1000 oflag-dsync
1000+0 records in
1000+0 records out
048###000 bytes (1.0 GB, 1000 MiB) copied, 10.8796 s, 96.4 MB/s

vSAN health checks indicated backend performance was optimal, isolating the issue to workload-level I/O characteristics.

Namespace object analysis confirmed presence of non-4K-aligned I/Os associated with snapshot metadata updates.

2025-10-06T03:34:07.082469 [49#####90] [cpu41] [71702924 OWNER readWriteAtsWithBlkAttr5 NSIO fullTrace] DOMTraceOpDiagnostic:10369: {'op': 0x45#######1c0, 'objUuid': '08####64-####-####-####-78a######15c', 'offset': 3735552, 'length-32': 512, 'timeInThisPhaseMS': 0, 'opPhase': 'DOM owner entire op'}
2025-10-06T03:34:12.080038 [49#####36] [cpu16] [fb20b93f OWNER readWriteAtsWithBlkAttr5 NSIO fullTrace] DOMTraceOpDiagnostic:10369: {'op':0x45#######1c0, 'objUuid': '08####64-####-####-####-78a######15c', 'offset': 3870720, 'length-32': 512, 'timeInThisPhaseMS': 1, 'opPhase': 'DOM owner entire op'}

Snapshots retained beyond 72 hours correlated with ongoing namespace updates and elevated latency events.

Resolution

• Change the vSAN storage policy for the affected database disks from RAID-5 to RAID-1 (FTT=1) to improve small I/O write performance.
• Avoid retaining snapshots for more than 72 hours to prevent non-aligned metadata writes on namespace objects.

Additional Information

vSAN Performance is degraded when I/Os are not 4k aligned

Troubleshooting vSAN Latency and Performance Issues