One or more virtual machine guest file systems are running out of disk space followed by multiple errors on edges.

Products

• VMware NSX
• VMware vSphere ESXi 8.0

Issue/Introduction

• High storage latencies observed on NSX clusters hosting edges.
• Edges clusters had an IO peak followed by high latency
• Logs were being generated from edges and manager
• There are no indications of significant host level processes or operations which would generate significant write I/O
• On vmkernel.log you observe IO failure for write command (0x2a) similar to:
  ScsiDeviceIO: 4121: Cmd(0x45d9a99924c8) 0x2a, CmdSN 0x800e002d from world xxxx to dev "naa.xxxxxxxxxxxxx" failed H:0x0 D:0x28 P:0x0

Environment

VMware ESXi 7.X
VMware ESXi 8.X
NSX-T

Cause

The combined IO of multiple VM's were exceeding threshold for IOPS and bandwidth of underlying storage device.

Resolution

Make sure there are enough available IOPS on storage device backing edge VM's and that latency is bellow thresholds as defined at: NSX Manager VM and Host Transport Node System Requirements

Additional Information

Slowness/latency reported on multiple VMs
NSX Manager VM and Host Transport Node System Requirements