vVol Datastore "Invalid argument" error: VM Power On or Folder Deletion fails

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article address an issue related to a state mismatch occurring within vVol datastores where Virtual Machines (VMs) fail to power on or cannot be deleted in vSphere 7.x and 8.x environments using Pure Storage or Nimble Storage. 

• VM placement fails with the error: Cannot complete file creation operation.

• "File Not Found" errors appear in the vSphere Client UI.

• When listing the contents of the VM directory from the ESXi command line, the host reports an Invalid argument error:
  
  

  [root@esxi_name:/vmfs/volumes/vvol:################-################] ls -ltrha VM_Name
  ls: VM_Name: Invalid argument

• When deleting a VM from the vCenter UI, the task fails with: Object was not found.

• When attempting to delete the VM folder via the ESXi command line, it reports an Invalid argument error:
  
  

  [root@esxi_name:/vmfs/volumes/vvol:################-################] rm -rf VM_Name
  rm: can't stat 'VM_Name': Invalid argument

• Orphaned Config vVols and Volume Groups remain on the storage array after a vRA deployment deletion task.
• The /var/log/vvold.log on the ESXi host repeatedly shows INVALID_ARGUMENT errors during bind operations:
  
  

  error vvold[3169182] [Originator@6876 sub=Default] SI:BindVirtualVolume@3467 VASA bind for VVol (rfc4122.17025cac-####-####-####-############) returned error : INVALID_ARGUMENT ()

 

Environment

• vSphere7.x, 8.x
• vCenter 7.x, 8.x
• Pure Storage FlashArray
• Nimble Storage

Cause

• There is a state mismatch where the array reports a vVol exists during the queryVirtualVolume operation, but the subsequent bindVirtualVolume operation fails with INVALID_ARGUMENT.

  This can be confirmed by the presence of the following consistent log patterns in /var/log/vvold.log:

  The array reports that the vVOL exists when queried:

  info vvold[3169169] [Originator@6876 sub=Default] VasaOp::QueryVirtualVolumeInt [#169390]: ===> Issuing 'queryVirtualVolume' to VP [Datastore_name:Connected (Outstanding 0/5)]
  info vvold[3169169] [Originator@6876 sub=Default] VasaOp[#169390] ===> FINAL SUCCESS queryVirtualVolume VP (Datastore_name) Container (863211ba-####-####-####-############) timeElapsed=7 msecs (#outstanding 0)
  info vvold[3169169] [Originator@6876 sub=Default] QueryVirtualVolumeInt vvolUuid = rfc4122.17025cac-####-####-####-############
  

  
  
  Attempting to bind this VVOL results in the INVALID_ARGUMENT error:

  info vvold[3169182] [Originator@6876 sub=Default] Came to SI::BindVirtualVolume: esxContainerId 863211ba-a4c6-####-b7e3-############ VVol Id rfc4122.17025cac-####-####-####-############ bindType Normal (isConfigVvol: true)
  info vvold[3169182] [Originator@6876 sub=Default] VasaOp::BindVirtualVolume [#169437]: ===> Issuing 'bindVirtualVolume' to VP [Datastore_name:Connected (Outstanding 0/5)]
  info vvold[3169182] [Originator@6876 sub=Default] VasaOp[#169437] ===> FINAL SUCCESS bindVirtualVolume VP (Datastore_name) Container (863211ba-####-####-####-############) timeElapsed=134 msecs (#outstanding 0)
  error vvold[3169182] [Originator@6876 sub=Default] SI:BindVirtualVolume@3467 VASA bind for VVol (rfc4122.17025cac-####-####-####-############) returned error : INVALID_ARGUMENT ()
  

 

Resolution

Workaround:

If an immediate fix is required to restore VM management, restart the vvold service on the affected ESXi host(s) to clear the stale VASA session:

/etc/init.d/vvold restart

 

Permanent Fix:

This issue is caused by the Storage Vendor's VASA provider implementation. Contact your storage vendor (Pure Storage or Nimble) and provide the log snippets above. Request an investigation into:

• Why the bind operation returns INVALID_ARGUMENT for an existing vVol.

• Recommendations for VASA provider firmware updates or service restarts on the storage controllers.