NFS shared storage volume getting full when executing vApp copy/move in Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• The NFS drive getting full frequently even for small vapp copy process.
• Freeing up some space by removing several files, the issue will appear again once attempted a new move/copy task.
• The task will fail to execute exports or download for the transfer on the NFS shared storage.
• OVA/OVF files failing to upload because of stale files on the NFS share.
• OVA upload file may fail with the errors below:
  
  Unable to create spooling directory /opt/vmware/vcloud-director/data/transfer/####-########-####-########c31d 
  
  AND/OR
  
  

  Could not create folder "file:///opt/vmware/vcloud-director/data/transfer/####-########-####-########c31d"

• The following error will stop Cloud Director performing a vCenter to vCenter direct cloning and fall back to the OVF transfer which consumes space on the transfer NFS shared storage::

Error in CheckClone_Task for vm [vcId=f<VCENTER_ID>, moref=<VM_MOREF>]. | requestId=<REQUEST_ID>,request=POST https://vcloud.example.com/api/catalog/CATALOG-ID/action/captureVApp,requestTime=1761136614144,remoteAddress=###.###.###.###:38204,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 39.0 vcd=<VCD_ID>,task=<TASK_ID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-####-############) activity=(com.vmware.vcloud.vdc.impl.CopyContentsActivity,urn:uuid:########-####-####-####-############) activity=(com.vmware.vcloud.vdc.impl.CreateStoredVmActivity,urn:uuid:########-####-####-####-############) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmNonBlockingActivity,urn:uuid:########-####-####-####-############) activity=(com.vmware.vcloud.fabric.storage.copy.impl.CopyVmActivity,urn:uuid:########-####-####-####-############)
com.vmware.ssdc.library.exceptions.GenericVimFaultException: Underlying system error: com.vmware.vim.binding.vim.fault.NotSupportedHost
vCenter Server task (moref: session[<SESSION_ID>) failed in vCenter Server '<VCENTER_NAME>' (<VCENTER_ID>).

Environment

VMware Cloud Director 10.6.1

Cause

This issue is caused by the fact that the NFS shared folder is not big enough in size to allow the vApp copy/move task to complete. This will leave stale file in the NFS shared storage which will not be automatically removed by Cloud Director.

To view the space available on the NFS transfer share, run the following command from any VMware Cloud Director cells:

df -h

Resolution

To help solving the issue you will need to size the NFS Shared storage volume to take into consideration the space that will be used during the exports and downloads during transfers, database backups, multicell log bundles, etc. See Preparing the Transfer Server Storage for Your VMware Cloud Director Appliance

Alternatively if it's not possible to increase the size of the existing NFS share, consider replacing the existing one with a new larger share as outlined in the documentation Replace the Transfer Server Storage for Your VMware Cloud Director Appliance.

To clean up the NFS Shared folder because of this issue, open a new case with the Broadcom VMware Cloud Foundation Support Team. For more information, see How to Submit a Support Request