Commvault backup fails with 'failed to collect disk block information' due to storage throttling

Products

• VMware vSphere ESXi

Issue/Introduction

Customers using Windows Server 2022 virtual machines (VMs) with large disk sizes (e.g., ~17 TB) on vSphere 8.x environments may experience the following symptoms:

• Commvault backups fail with the specific error: failed to collect disk block information.
• Snapshot removal tasks hang indefinitely, leading to VM unresponsiveness.
• Datastore browsing in vCenter Server times out or becomes extremely slow.
• VM power-on operations stall consistently at 66%.
• ESXi host logs (vmkernel.log) may not show standard storage latency or path degradation indicators.

Environment

ESXi 8.x

Cause

This issue is typically caused by a combination of infrastructure-level constraints and stale file locks:

1. Storage IO Throttling: An IO throttle limit enforced at the array level (e.g., Nimble Storage) restricts throughput to the volume. This starvation prevents the ESXi host from completing high-IO operations like snapshot consolidation and metadata updates during backups. VM backup getting slow
2. Stale File Locks: A secondary issue occurs when an interactive shell process (such as a less or cat command) holds an active cwd (current working directory) lock on the VM namespace. This lock prevents the ESXi host from properly enumerating virtual disks, causing the power-on task to stall at 66% Unable to enumerate all disks.

Resolution

1. Resolve Storage Throttling

Engage your storage vendor (e.g., Nimble Storage support) to verify and remove IO throttle limits on the affected volume. Restoring required IOPS throughput is necessary for backup and consolidation tasks to complete.

2. Clear Stale File Locks

Identify and terminate any processes holding a lock on the VM directory.

 IMPORTANT: Always test command updates first in a non-production environment. Take a backup before implementing in production.

• Log in to the ESXi host via SSH.
• Run the following command to find the process ID (PID) holding the lock:
  bash

   

  vmkvsitools lsof | grep <VM_NAME_OR_UUID>

• If a less or sh process is identified holding a cwd lock, terminate the PID Unable to enumerate all disks
  bash

   

  kill -9 <PID>

3. Verify and Retry

• Verify the locks are cleared by querying the VMDK:
  bash

   

  vmkfstools -q /vmfs/volumes/<DATASTORE>/<VM_NAME>/<VM_NAME>.vmdk -v10

• Retry the VM power-on and snapshot consolidation tasks in vCenter Server.

Additional Information

Best practices for using VMware snapshots

Investigating virtual machine file locks