When using all NVMe devices on VMware VSAN with deduplication, during large bursts of I/O, write buffers can fill and introduce congestion causing latency through the environment.
VMware vSAN
With deduplication, vSAN is only able to de-stage data as fast as the software allows it, this is because the deduplication process takes place during de-staging. Paired with using high performance devices, this can result in a situation where I/O is hitting the write buffers faster than data can be de-staged, causing a bottleneck that results in SSD congestion and cluster wide latency.
Consider turning off deduplication and using compression only for space savings.
When using high performance devices, it's not recommended to use Deduplication as well so you can avoid bottlenecking large bursts of I/O at the de-stage process.