Storage vMotion and Virtual Machine Creation Fails in NFS Datastore Cluster with Storage DRS Enabled

Products

• VMware vCenter Server

Issue/Introduction

Symptoms :

• The following error is noticed while deploying the new VM under the datastore cluster.

• Storage vMotion of VM to this datastore cluster will also fail.

 

Validation:

• VMs can be deployed from the host client to the datastore.

• VM can be deployed on other datastore.

• VM can be deployed if the storage DRS is disabled on this impacted datastore cluster.

Error noticed while deploying the VM:

Error from the vCenter, vpxd logs :

2025-02-25T10:08:56.106+01:00 info vpxd[08699] [Originator@6876 sub=vmomi.soapStub[269] opID=vcd-a4eb16b7-cacc-44dc-bccf-5e81ff03cc04-3] SOAP request returned HTTP failure; <<io_obj p:0x00007fa44cab6bf8, h:115, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-432852/vpxa>, method: acquireTicket; code: 500(Internal Server Error); fault: (vim.fault.InvalidState) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007fa44cab6bf8, h:115, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-432852/vpxa>]: acquireTicket
--> Received SOAP response fault from [<<io_obj p:0x00000012d04430c0, h:18, <TCP '127.0.0.1 : 34385'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: acquireTicket
--> The operation is not allowed in the current state."
--> }


Verifying the error from the Web Developer tool :

1. Open the vCenter

2. On vCenter page, Open Browser "Web developer tools" / press F12 on keyboard.

3. Navigate to tab > Network > Response

4. Initiate the create VM task on the vCenter.

5. When the create VM task fails/errors. -> switch to web developer tools page.

6. Navigate to API "propertyObjectType=com.vmware.vsphere.client.vm.VmCreateSpec "

7. Select Response on the right side tab to see the below error:

Error : "message": "The operation is not allowed in the current state of the datastore. \nCould not migrate VM to the Datastore, since the number of poweredOn VMs limit is reached.",

Environment

VMware vCenter Server 7.0
VMware vCenter Server 8.0

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

• This issue occurs when the number of powered-on virtual machines per datastore has exceeded the limit.

• Storage DRS prevents virtual machines from powering on in excess of the host-based limitation of powered-on VMX files per datastore (2,048).

Resolution

Since we have exceeded the powered-on limit on the datastore cluster, there is no room to deploy any new VM or migrate VMs to this datastore cluster. This leaves us with only two options:

1. Add New datastore to this datastore cluster.

or 

2. Disable the Storage DRS on this datastore cluster.