Backups affected by delayed or lengthy Datastore Browser directory listing

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

You may encounter delays when reviewing virtual machine directory content in the Datastore Browser of a VMware VirtualCenter Server or vCenter Server product.

• Completing a directory listing of the virtual machine's directory through the Datastore Browser takes long time
• Connecting a VMware Infrastructure or vSphere Client directly to the ESX Server running the virtual machine does not experience significant delays
• Connecting a VMware Infrastructure or vSphere Client to any alternate ESX Server (not running the virtual machine ) experience the delays
• Browsing the directory directly using a console or SSH session shows no observable delays

Environment

VMware ESX 4.0.x
VMware ESXi 3.5.x Embedded
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable
VMware ESX Server 3.0.x
VMware ESXi 3.5.x Installable
VMware vCenter Server 4.0.x
VMware vCenter Server 4.1.x
VMware ESX 4.1.x
VMware ESX Server 3.5.x
VMware VirtualCenter 2.5.x
VMware ESXi 4.0.x Installable

Resolution

When VMware ESX starts a virtual machine, it creates file locks against certain virtual machine files. These locks are maintained throughout a virtual machine's runtime until it is stopped or powered-off.

Browsing datastores in the VMware Infrastructure or vSphere Client invokes "File Open" (or fopen) operations against stored files in a given datastore directory.

If these fopen operations are invoked from the VMware ESX host running the virtual machine, the resulting datastore directory list is completed almost immediately. However if other VMware ESX servers (otherwise those not running the virtual machine) attempt the same operation, it takes considerably longer.

These hosts do not have the ability to access the locked files for the requested data, requiring the operations to time out. This induces delays which by themselves are insignificant, however cumulatively they may cause a perceivable or significant delay before receiving a completed file listing in the Datastore Browser.

The Datastore Browser included with VMware VirtualCenter Server and vCenter Server does not include internal logic to determine which host is maintaining a given virtual machine's locks. As a result, the selected host to perform file listings is not guaranteed to be the current virtual machine lock owner and may encounter delays.

Using the VMware API

The Browse Datastore operation may still be utilized by a backup product. It is possible to avoid these delays by embedding logic in the backup software or scripts to prevent the delays that may be imposed:

• Determine the current VMware ESX server of a running virtual machine, whose files are locked by this host server and will eventually require looking-up for backup.
• Once the ESX Server has been determined, the file search query should be directed at that particular host, as opposed to the VMware VirtualCenter or vCenter Server, or any other alternate ESX Server.
• Confirm that the file list gathering process completes in a timely manner.
• Use a custom script that takes on the responsibility of determining the target host.

Note: This logic should only be applied to powered-on virtual machines; this issue does not apply to powered-off or otherwise lock-free virtual machine files.

Logic Breakdown

• For a given virtual machine object, locate the property called runtime, of the type: VirtualMachineRuntimeInfo.
• This data object contains a property named host which is of the type: HostSystem.
• Upon retrieving the host object from RuntimeInfo, the name property containing the hostname of the configured VMware ESX host can be found.