Disk consolidation reports "Unknown error" due to VAAI plugin memory exhaustion

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Below error will be seen when trying to perform VM Consolidation
  

  "An error occurred while consolidating disks: Unknown error."

• vmware.log  report the following error for snapshot consolidation:
  
  YYYY-MM-DDTHH:MM:SS.MSZ Er(02) vmx - DISKLIB-LINK  : DiskLinkGetSize: Failed to get extent size: Unknown error
YYYY-MM-DDTHH:MM:SS.MSZ  vmx - ConsolidateFillSnapDiskTransferArray: Failed to fetch disk size of '/vmfs/volumes/########-########-####-############/<VM-name>/#######-000001.vmdk' for consolidate: Unknown error (5)
YYYY-MM-DDTHH:MM:SS.MSZ  vmx - ConsolidateStart: Failure while populating the snapDataPerCombineOperation array: 'Unknown error' (5)
YYYY-MM-DDTHH:MM:SS.MSZ  vmx - VigorTransport_ServerSendResponse opID=#########-##-#### seq=#######: Completed Snapshot.Consolidate request with messages in 7673 US.
YYYY-MM-DDTHH:MM:SS.MSZ  vmx - Turning off snapshot info cache.
YYYY-MM-DDTHH:MM:SS.MSZ  vmx - Turning off snapshot disk cache.
YYYY-MM-DDTHH:MM:SS.MSZ  vmx - ConsolidateEnd: Snapshot consolidate complete: Unknown error (5).

• In /var/run/log/vmkernel.log , nasd reports no memory available
  
  

  YYYY-MM-DDTHH:MM:SS.MSZ In(182) vmkernel: cpu12:#######)Admission failure in path: host/vim/vmvisor/vaai-nasd:vaai-nasd.#######:pt.#######
YYYY-MM-DDTHH:MM:SS.MSZ In(182) vmkernel: cpu12:#######)pt.####### (6390) requires 4 KB, asked 4 KB from vaai-nasd (895) which has 40960 KB occupied and 0 KB available.
YYYY-MM-DDTHH:MM:SS.MSZ In(182) vmkernel: cpu12:#######)Admission failure in path: host/vim/vmvisor/vaai-nasd:vaai-nasd.#######:pt.#######
YYYY-MM-DDTHH:MM:SS.MSZ In(182) vmkernel: cpu12:#######)pt.####### (6390) requires 4 KB, asked 4 KB from vaai-nasd (895) which has 40960 KB occupied and 0 KB available.

Environment

• VMware vCenter 8.x
• vSphere ESXi 8.x

Cause

The issue occurs due to memory exhaustion within the VAAI NAS daemon (vaai-nasd) on the ESXi host; because of a known memory leak condition, the vaai-nasd plugin consumes all allocated memory over time and is left with no available memory to service new requests.

During snapshot consolidation, ESXi must query snapshot disk metadata (such as extent size and disk layout) through the VAAI NAS layer; since the vaai-nasd process has no free memory available, these metadata queries fail, causing disk size retrieval to fail and the consolidation operation to abort with a generic “Unknown error”.

 

Resolution

• Restart following daemon on affected ESXi:
  
  # /etc/init.d/vaai-nasd stop
# /etc/init.d/vaai-nasd start
  
  
• Workaround: reboot the affected host to fully reset the vaai-nasd process and reclaim memory.
  
  
• Please engage the storage vendor to investigate further on the memory leaks.