Starting virtual machines with multiple VMDKs running on an NFS is very slow after a HA failover

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• After a HA failover, it takes a long time for virtual machines with multiple VMDKs running on an NFS to boot
• In the VMware.log file, you see entries similar to:
  
  | vmx| DISK: OPEN scsi0:0 '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3.vmdk' persistent R[]
  | vmx| DISKLIB-VMFS : "/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3-flat.vmdk" : open successful (10) size = 4294967296, hd = 54244. Type 3
  | vmx| DISKLIB-DSCPTR: Opened [0]: "NFS_VM3-flat.vmdk" (0xa)
  | vmx| DISKLIB-LINK : Opened '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3.vmdk' (0xa): vmfs, 8388608 sectors / 4 GB.
  | vmx| DISKLIB-LIB : Opened "/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3.vmdk" (flags 0xa, type vmfs).
  | vmx| DISK: Disk '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3.vmdk' has UUID '60 00 c2 92 6d b6 40 54-fb 03 88 f2 bb 48 71 1a'
  | vmx| DISK: OPEN '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3.vmdk' Geo (522/255/63) BIOS Geo (0/0/0)
  | vmx| DISK: OPEN scsi0:1 '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_1.vmdk' persistent R[]
  | vmx| DISKLIB-VMFS : "/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_1-flat.vmdk" : open successful (10) size = 8589934592, hd = 341427. Type 3
  | vmx| DISKLIB-DSCPTR: Opened [0]: "NFS_VM3_1-flat.vmdk" (0xa)
  | vmx| DISKLIB-LINK : Opened '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_1.vmdk' (0xa): vmfs, 16777216 sectors / 8 GB.
  | vmx| DISKLIB-LIB : Opened "/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_1.vmdk" (flags 0xa, type vmfs).
  | vmx| DISK: Disk '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_1.vmdk' has UUID '60 00 c2 92 a6 6d 1f 4d-62 a8 f8 4b 0e 88 1d 79'
  | vmx| DISK: OPEN '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_1.vmdk' Geo (1044/255/63) BIOS Geo (0/0/0)
  | vmx| DISK: OPEN scsi0:2 '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_2.vmdk' persistent R[]
  | vmx| DISKLIB-VMFS : "/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_2-flat.vmdk" : open successful (10) size = 8589934592, hd = 331168. Type 3
  | vmx| DISKLIB-DSCPTR: Opened [0]: "NFS_VM3_2-flat.vmdk" (0xa)
  | vmx| DISKLIB-LINK : Opened '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_2.vmdk' (0xa): vmfs, 16777216 sectors / 8 GB.
  | vmx| DISKLIB-LIB : Opened "/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_2.vmdk" (flags 0xa, type vmfs).
  | vmx| DISK: Disk '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_2.vmdk' has UUID '60 00 c2 9e f4 4c 00 4a-23 2f 82 ab c8 8b 31 03'
  | vmx| DISK: OPEN '/vmfs/volumes/4978db5f-76fb8163/NFS_VM3/NFS_VM3_2.vmdk' Geo (1044/255/63) BIOS Geo (0/0/0)
   
• The log entries indicate that each VMDK file is opened serially and it takes 40 seconds to open each file

Environment

VMware ESXi 4.1.x Installable
VMware ESX 4.1.x
VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Embedded

Resolution

In ESX/ESXi 4.1 and 5.0, the VMDK files are opened serially and this is the default behavior.

The time it takes to open a disk is controlled by the NFS Locking advanced parameters. For more information, see Definition of the advanced NFS options (1007909).

In ESXi 5.1, the behavior is changed and disks are opened in parallel. ESX 5.1 supports five VMDK disks to be opened in parallel at a time.