Apply changes fails due to cpi error: No space left on device
Article ID: 293738
Updated On:
Products
Operations Manager
Issue/Introduction
VM creation during upgrade or deployments fails with vSphere CPI error:
The issue comes from datastore placement code more or less randomly picks from the datastores that do have enough room place the VM at the time the calculation is made. This calculation is made without considering any other in-flight VM creation operations. Simultaneous VM creation is done by the Bosh Director starting up multiple, entirely independent vSphere CPI instances. (One CPI instance is started per VM that is going to be created.) These CPI instances operate entirely independently and do not communicate with each other, so it’s entirely possible that multiple independent CPI instances will choose to put VMs in a Datastore that has room for one but not more VMs, and either VM creation or VM startup of some or all of those VMs will fail.
'Module MonitorLoop power on failed. ,Failed to extend swap file /vmfs/volumes/uuid_xxxx.vswp from 0 KB to xxxx KB : No space left on device,Current swap file size is 0 KB.,Failed to extend the virtual machine swap file,Could not power on virtual machine: No space left on device.,Failed to power on VM.,Failed to start the virtual machine.' ERROR -- [req_id cpi-876582]: Error in creating vm: Module MonitorLoop power on failed.Platform may have other datastores available with enough space to place the VM.
The issue comes from datastore placement code more or less randomly picks from the datastores that do have enough room place the VM at the time the calculation is made. This calculation is made without considering any other in-flight VM creation operations. Simultaneous VM creation is done by the Bosh Director starting up multiple, entirely independent vSphere CPI instances. (One CPI instance is started per VM that is going to be created.) These CPI instances operate entirely independently and do not communicate with each other, so it’s entirely possible that multiple independent CPI instances will choose to put VMs in a Datastore that has room for one but not more VMs, and either VM creation or VM startup of some or all of those VMs will fail.
Environment
Product Version: 2.10
Resolution
To guarantee that you'll never hit this error, you will need to have free space equal to the size of all of the disks and all of the RAM of all VMs that could be deployed simultaneously by the Bosh Director.
Alternatively, reducing the max_in_flight and max_threads to 1 would also avoid this issue.
Opsmanager 2.10.56 and 3.0.7 have vSphere CPI version 91 which has a fix for the datastore placement mechanic issue.
Alternatively, reducing the max_in_flight and max_threads to 1 would also avoid this issue.
Opsmanager 2.10.56 and 3.0.7 have vSphere CPI version 91 which has a fix for the datastore placement mechanic issue.
Feedback
Yes
No
Powered by
