Extremely slow read throughput during Commvault NBD backups on ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

This article addresses an issue where virtual machine backups using the Network Block Device (NBD) transport mode experience severe performance degradation, specifically characterized by extremely low read speeds.

Symptoms:

• Backup read speeds drop as low as 0.005 MB/sec for virtual machines on a specific host. Review of the Commvault vsbkp.log shows below entries.

  12956 2208   07/01 12:04:12 11730629 stat- ID  [readdisk], Bytes [312475977], Time [53659. 726612] Sec(s), Average Speed [0.005554] MB/Sec
  12956 2208   07/01 12:04:12 11730629 stat- ID  [Datastore Read [<Datastore_name>]], Bytes [1594884775], Time [53689.737842] Sec(s), Average Speed [0.028329] MB/Sec

• NFC sessions are established successfully but stall or cease data transfer after a period of time.

• Backups may run for 15+ hours before being manually canceled.

• The issue is localized to specific network paths between a particular Backup Media Agent and ESXi host.

• The backup media agent is a virtual server agent running on a Windows Virtual Machine.

• ESXi vmkernel.log and hostd.log shows successful snapshot creation and NFC handshake completion without errors.

• Alternative Media Agents on different network segments are able to back up the same VM at normal speeds.

• The affected Media Agent can back up virtual machines on other hosts successfully, indicating the bottleneck is unique to the specific route between the agent and the target host.

Environment

• Product: VMware vSphere ESXi 8.0 Update 3
• Backup Software: Commvault (using NBD transport mode)
• Network Configuration: Single VMkernel adapter shared for both Management and vSphereBackupNFC traffic with 1 GBit/s physical uplinks

Cause

The underlying cause is an external network transit path bottleneck unique to the route between the virtual Media Agent and the ESXi management interface . 

Host-side stability is confirmed when:

• Snapshots are created and consolidated successfully without errors in hostd.log.
• NFC handshakes succeed, but incoming read requests cease after the initial connection.
• Alternative Media Agents on different network segments complete backups successfully for the same virtual machines.

Resolution

To resolve the performance bottleneck, follow these steps:

1. Isolate Backup Traffic (Best Practice): Create a dedicated VMkernel adapter for backup traffic to prevent contention with management traffic.

   • In the vSphere Client, navigate to the ESXi Host > Configure > Networking > VMkernel adapters.
   • Click Add Networking and select VMkernel Network Adapter.
   • Select an existing switch or create a new one.
   • Under Available services, select vSphere Backup NFC.
   • Assign a dedicated VLAN and IP subnet if possible to segregate this traffic from the Management network.

2. Verify Network Path Integrity: Engage the networkteam to perform a bi-directional path trace between the Media Agent IP and the ESXi VMkernel IP.

   • Check for interface errors (CRC, drops) on intermediate physical switches.
   • Verify if MTU mismatches (e.g., Jumbo Frames misconfiguration) exist along the path.

Workaround

If an immediate fix for the network bottleneck is not available, modify the backup selection policy:

• Assign a different Media Agent (one known to have a clean network path to the affected host) to handle the backups for VMs residing on that specific host.