Performing cold migration to other datastores or cloning of a virtual machine fails when using NetApp NFS storage in cluster-mode with VAAI enabled

Products

• VMware vSphere ESXi

Issue/Introduction

• Performing a cold migration to other datastores, or cloning of a virtual machine fails in ESXi host 5.5 Update 1 when using NetApp FAS6220 array with ONTAP 8.2.1 NFS storage in cluster-mode with VAAI enabled.
• In vSphere, you see this error:
  
  Error caused by file /vmfs/volumes/<UUID>/<VM>/<VMDKNAME>.vmdk
   
• The virtual machine is powered off and there are no lock files in the folder of the virtual machine. Disk files (.vmdk) are in a Normal state.
• Performing Storage vMotion (when the virtual machine is in a powered on state) and copying files over the datastore browser to another datastore works.
• This issue does not occur in NetApp NFS in 7-mode.
• Hardware acceleration is in "Supported" status on the NFS datastore.
• In the hostd log file (on the ESXi host), located at /var/run/log/, you can see entries similar to:
  
  <YYYY-MM-DD><time> [4EEC2B70 error 'Vmsvc.vm:/vmfs/volumes/<UUID>/<VM>/<VM>.vmx' opID=hostd-xxxx user=vpxuser] Exception Unknown error trying to get sizes from /vmfs/volumes/<UUID>/<VM>/<VMDKNAME>.vmdk
  <YYYY-MM-DD><time> [4EEC2B70 error 'Vmsvc.vm:/vmfs/volumes/<UUID>/<VM>/<VM>.vmx' opID=hostd-xxxx user=vpxuser] Exception Unknown error trying to get sizes from /vmfs/volumes/<UUID>/<VM>/<VMDKNAME_1>.vmdk
  <YYYY-MM-DD><time> [4EEC2B70 info 'Hostsvc' opID=hostd-xxxx user=vpxuser] Lookupvm: World ID not set for VM 1
  <YYYY-MM-DD><time> [4EEC2B70 verbose 'Hbrsvc' opID=hostd-xxxx user=vpxuser] Replicator: VmFileProviderCallback VM (id=1)
  
  </time></time></time></time></time></time>
• In the vmkernel log file, located at /var/log/, you can see entries (VAAI related issues) similar to:
  
  <YYYY-MM-DD><time> [4EEC2B70 info 'Libs' opID=hostd-xxxx user=vpxuser] VAAI-NAS :: vendor_nas_plugin: SUCCESSES: RsrvSpace [0] Cln-Full
  [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [2081]
  <YYYY-MM-DD><time> [4EEC2B70 info 'Libs' opID=hostd-xxxx user=vpxuser] VAAI-NAS :: vendor_nas_plugin: FAILURES: RsrvSpace [0] Cln-Full [
  0] Cln-Lazy [0] cln-DRun [0], Ext-stats [43577]
  <YYYY-MM-DD><time> [4EEC2B70 info 'Libs' opID=hostd-xxxx user=vpxuser] VAAI-NAS :: SvaNasPlugin: SUCCESSES: RsrvSpace [0] Cln-Full [0] C
  ln-Lazy [0] cln-DRun [0], Ext-stats [0]
  <YYYY-MM-DD><time> [4EEC2B70 info 'Libs' opID=hostd-xxxx user=vpxuser] VAAI-NAS :: SvaNasPlugin: FAILURES: RsrvSpace [0] Cln-Full [0] Cl
  n-Lazy [0] cln-DRun [0], Ext-stats [0]
  <YYYY-MM-DD><time> [4EEC2B70 info 'Libs' opID=hostd-xxxx user=vpxuser] VAAI-NAS :: vmfsNasPlugin: SUCCESSES: RsrvSpace [0] Cln-Full [0]
  Cln-Lazy [0] cln-DRun [0], Ext-stats [0]
  <YYYY-MM-DD><time> [4EEC2B70 info 'Libs' opID=hostd-xxxx user=vpxuser] VAAI-NAS :: vmfsNasPlugin: FAILURES: RsrvSpace [0] Cln-Full [0] C
  ln-Lazy [0] cln-DRun [0], Ext-stats [0]
  <YYYY-MM-DD><time> [4EEC2B70 info 'Libs' opID=hostd-xxxx user=vpxuser] VAAI-NAS :: NAS Mapping Used successfully for 47153 times</time></time></time></time></time></time></time>

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0

Resolution

Note: This is not a VMware software issue. This solution is not directly supported by VMware. For issues with this configuration, contact NetApp directly to change the configuration in the array.

 

To work around this issue, change the configuration in the array as NFSv3 and v4 are required for proper functioning of NFS VAAI.

Additionally, to allow VAAI calls, the NFSv4 protocol needs to be allowed.

 

To allow the NFSv4 protocol, use one of these options:

• Set the allowed protocol in the rule to NFS (that encompasses all versions of NFS).
• Set it to NFSv3 or NFSv4 to include both version 3 (for the datastore access) and NFSv4 for the VAAI specific calls.

 

Note: Make sure to enable NFSv4 in the NetApp array only. ESXi hosts are still using NFSv3 (Currently, NetApp only supports NFSv3 for ESXi 5.5 Update 1).