Backup speed is slow over NBD transport mode for VMs on high-latency storage

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides a way for backup vendor to improve the backup performance for high-latency storage.

Symptoms:

Customer is using a storage with about 4ms latency for each IO.

When using VDDK NBD/NBDSSL transport mode, backup throughput is quite slow even for single stream.
In a customer case that average latency was 4ms, we saw 15 MB throughput for single backup stream.

 

Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 6.7

Cause

There is high latency for storage IO.

The default AIO manager in NFC Server (which VDDK connects to in NBD mode) doesn’t have many threads to handle IOs asynchronously.

Resolution

Option 1: Upgrade Storage Array where latency is minimal.

Option 2: Utilize VIXDISKLIB_FLAG_OPEN_UNBUFFERED and NFC AIO configurable options


A. open disk with open flag VIXDISKLIB_FLAG_OPEN_UNBUFFERED to make IO handled by more efficient AIO manager in ESXi host.

1) Specify open flag VIXDISKLIB_FLAG_OPEN_UNBUFFERED when NBD mode is used to open (VixDiskLib_Open) the disk
2) Issue IOs: minimum 4KB alignment and a multiple of 4KB as IO size

    The performance may increase up to 4x if ESXi hosts are vSphere 7.0 U2 and later.

B. Configurable NFC AIO buffer options since VDDK 7.0 u3 and VDDK 6.7 u3 EP1

Users can configure asynchronous I/O buffers for NBD(SSL) transport. Defaults are buffer size 64K (1) and buffer count 4. In testing, these InitEx configuration file settings performed best, but this depends on hardware setup:
vixDiskLib.nfcAio.Session.BufSizeIn64KB=16
vixDiskLib.nfcAio.Session.BufCount=4

Note:

1) it’s highly recommended the VDDK IO buffer multiple of vixDiskLib.nfcAio.Session.BufSizeIn64KB to archive better performance.
2) Memory consumption in NFC server will be increased with larger NFC AIO buffer size and buffer count.

Refer to VDDK programming guide for more information.

Workaround:
No Workaround