A virtual machine fails to power on with the error: Unexpected errno=87, Too many users

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

When using virtual machines which mount the same RDM disk with many virtual machines running on many hosts, using Physical Compatibility mode in ESXi/ESX, you experience these symptoms:

When 8 virtual machines share the RDM, and reside on 8 ESXi/ESX hosts, all virtual machines successfully power on
If any virtual machine sharing the RDM is powered on or migrated to a 9th ESXi/ESX host, this virtual machine fails to power on
A virtual machine fails to power on with the error:

Unexpected errno=87, Too many users
In the virtual machine log file (vmware.log), you see entries similar to:

2015-07-29T01:55:16.368Z| vmx| I120: FileIOErrno2Result: Unexpected errno=87, Too many users

2015-07-29T01:55:16.368Z| vmx| I120: AIOGNRC: Failed to open '/vmfs/volumes/556668a8-31038112-6d2b-2cd444f152fe/xxxx/xxxx-rdmp.vmdk' : Too many users (570002) (0x4013).

2015-07-29T01:55:16.368Z| vmx| I120: OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/556668a8-31038112-6d2b-2cd444f152fe/xxxx/xxxx-rdmp.vmdk' : Too many users (5701636).

2015-07-29T01:55:16.368Z| vmx| I120: DISKLIB-VMFS : "/vmfs/volumes/556668a8-31038112-6d2b-2cd444f152fe/xxxx/xxxx-rdmp.vmdk" : failed to open (Too many users): ObjLib_Open failed. Type 10

2015-07-29T01:55:16.368Z| vmx| I120: DISKLIB-LINK : "/vmfs/volumes/556668a8-31038112-6d2b-2cd444f152fe/xxxx/xxxx.vmdk" : failed to open (Too many users).

2015-07-29T01:55:16.368Z| vmx| I120: DISKLIB-CHAIN : "/vmfs/volumes/556668a8-31038112-6d2b-2cd444f152fe/xxxx/xxxx.vmdk" : failed to open (Too many users).

2015-07-29T01:55:16.368Z| vmx| I120: DISKLIB-LIB : Failed to open '/vmfs/volumes/556668a8-31038112-6d2b-2cd444f152fe/xxxx/xxxx.vmdk' with flags 0x4a Too many users (5701641).

2015-07-29T01:55:16.368Z| vmx| I120: DISK: Cannot open disk "/vmfs/volumes/556668a8-31038112-6d2b-2cd444f152fe/xxxx/xxxx.vmdk": Too many users (5701641).

2015-07-29T01:55:16.368Z| Worker#0| I120: FileIOErrno2Result: Unexpected errno=87, Too many users

2015-07-29T01:55:16.369Z| Worker#0| I120: AIOGNRC: Failed to open '/vmfs/volumes/556668a8-31038112-6d2b-2cd444f152fe/xxxx/xxxx_1-rdmp.vmdk' : Too many users (570002) (0x4013).

2015-07-29T01:55:16.369Z| Worker#0| I120: OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/556668a8-31038112-6d2b-2cd444f152fe/xxxx/xxxx_1-rdmp.vmdk' : Too many users (5701636).

Note: The preceding log excerpts are examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Embedded
VMware ESX 4.0.x
VMware ESX Server 3.5.x
VMware ESXi 3.5.x Embedded
VMware ESXi 3.5.x Installable
VMware vSphere ESXi 5.1
VMware vSphere ESXi 6.0
VMware ESX 4.1.x
VMware vSphere ESXi 5.5
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Installable

Cause

VMFS clustered locking uses an on-disk structure disk lock type that is used to synchronize concurrent access to shared resources (files, folders and other metadata structures) from multiple ESXi/ESX hosts connected to a SAN.

The clustered lock manager supports three different types of locks: exclusive, read-only and multi-writer.

Exclusive locks permit only one host at a time to access the associated resource
Read-only locks permit concurrent read-only access by multiple hosts
Multi-writer locks permit concurrent writes from multiple hosts

The file system tracks the identity of all hosts currently accessing the resource. The disk lock structure has space for 8 host identities.

The same restrictions also apply to these file sharing use cases:

SE Sparse based snapshots/linked clones
Multi-writer lock uses case like Oracle RAC solutions that share a data/quorum disk.
CD and floppy images attached to virtual machines deployed on 9 or more hosts
RDM mounted to a virtual machine deployed on more than 8 hosts

Note: On ESXi 5.x and later with VMFS-5 volumes, share read-only access to virtual disks with up to 32 hosts (for View, linked clones, and fast provisioning use cases).