Runtime fault vmodl.fault.SystemError with reason 'No such file or directory' seen on invoking DatastoreBrowser.searchSubFolders task.

Products

• VMware vSphere ESXi

Issue/Introduction

We have seen this behavior when concurrent delete and searchSubFolders operations are performed on a datastore. This issue can occur more frequently if the underlying storage is slow or has a huge number of entries (for example greater than 40K). This issue occurs because of a race condition between delete and search operations. Customers should avoid such concurrent delete and search operations if they see this issue.

Symptoms:

Specific error message:

(vmodl.fault.SystemError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'A general system error occurred: No such file or directory',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   reason = 'No such file or directory'
}

This is a runtime fault thrown from OS layer. There is no specific filename/log entry for this.

Environment

VMware vSphere ESXi 6.7

Cause

The DatastoreBrowser.searchSubFolders task performs a search on a given datastore path recursively. During this process, if some other process deletes directories in this path, searchSubFolders can fail with 'No such file or directory' fault due to the conflict. This issue can occur more frequently if the underlying storage is slow or has a huge number of entries. The issue manifests when the VMware APIs for these tasks are used inefficiently, and should not be considered a VMware product bug.

Resolution

VMware is evaluating making this API more robust for such use cases, but this issue cannot be completely eliminated unless the above limitations are taken into consideration when using the API. VMware is also considering adding more details to the error messages based on future scenarios & use-cases.

Workaround:

Customers should avoid concurrent delete and searchSubFolders operations if they are seeing this issue.