Shared-nothing vMotion of a VM to another cluster fails with error 'The available Memory resources in the parent resource pool are insufficient for the operation'
Article ID: 315464
Updated On:
Products
Issue/Introduction
To be able to live migrate a VM from one cluster to another.
Symptoms:
Shared-nothing(both compute and storage vMotion) vMotion of a VM to another cluster fails with error 'The available Memory resources in the parent resource pool are insufficient for the operation'
vpxd.log
2023-12-11T06:08:23.178Z warning vpxd[10143] [Originator@1234 sub=obn opID=<opID>] post migrate callback is skipped - VM migration failed
2023-12-11T06:08:23.178Z error vpxd[10143] [Originator@1234 sub=vmpro999 opID=<opID>] Get exception while executing action vpx.vmprov.InvokeCallbacks: N3Vim5Fault32InsufficientMemoryResourcesFault9ExceptionE(Fault cause: vim.fault.InsufficientMemoryResourcesFault
--> )
2023-12-11T06:08:23.184Z info vpxd[10143] [Originator@1234 sub=vmpro999 opID=<opID>] Workflow context:
2023-12-11T06:08:23.230Z info vpxd[10143] [Originator@1234 sub=Default opID=<opID>] [VpxLRO] -- ERROR task-2069600 -- vm-12345 -- vim.VirtualMachine.relocate: vim.fault.InsufficientMemoryResourcesFault:
--> Result:
--> (vim.fault.InsufficientMemoryResourcesFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> unreserved = 1393831182336,
--> requested = 1125899906318336
--> msg = ""
--> }
--> Args:
-->
--> Arg spec:
--> (vim.vm.RelocateSpec) {
--> service = (vim.ServiceLocator) null,
--> folder = <unset>,
--> datastore = 'vim.Datastore:777776f8-b99c-4230-a820-7f0ea1311111:datastore-22222',
--> diskMoveType = <unset>,
--> pool = 'vim.ResourcePool:777776f8-b99c-4230-a820-7f0ea1311111:resgroup-11111',
--> host = 'vim.HostSystem:777776f8-b99c-4230-a820-7f0ea1311111:host-33333',
--> disk = (vim.vm.RelocateSpec.DiskLocator) [
--> (vim.vm.RelocateSpec.DiskLocator) {
Environment
Cause
The large required value is OMM_MAX_OVERHEAD_BYTES, which is set when we do not have a valid memory overhead limit for a vm/host pair.
Based on the vpxd logs, it appears OMM could not parse the vmxConfig for vm-12345 successfully due to spaces in some custom keys in the vmdk descriptor.
2023-12-11T05:45:28.035Z warning vpxd[07699] [Originator@1234 sub=OMM opID=<opID>] ConfigParser: parse error near key #disk0.ddb.customVolumeMetadataName\\Hard
2023-12-11T05:45:28.035Z warning vpxd[07699] [Originator@1234 sub=OMMVmInfo opID=<opID>] Failed to parse vmxConfig for VM [vim.VirtualMachine:vm-12345
Looking at the vmdk files for this VM in esx log bundle, we see these key/value pairs with spaces in the key in all 3 vmdks.
disk4000.vmdk
ddb.customVolumeMetadataName\\Hard disk 1\Volume 1 System Reserved = "77777##########12345:6705"
ddb.customVolumeMetadataName\\Hard disk 1\Volume 2 = "88888#########54321:6705"
disk4001.vmdk
ddb.customVolumeMetadataName\\Hard disk 2\Volume 1 Pagefiles = 999999#########11111:6705"
disk4002.vmdk
ddb.customVolumeMetadataName\\Hard disk 3\Volume 1 New Volume = "666666########9456:6705"
Resolution
1. Take a backup of the VM/take a copy of the files to be modified.
2. Shut down the VM.
3. Remove the below lines from the respective files.
> disk4000.vmdk
ddb.customVolumeMetadataName\\Hard disk 1\Volume 1 System Reserved = "77777##########12345:6705"
ddb.customVolumeMetadataName\\Hard disk 1\Volume 2 = "88888#########54321:6705"
> disk4001.vmdk
ddb.customVolumeMetadataName\\Hard disk 2\Volume 1 Pagefiles = 999999#########11111:6705"
> disk4002.vmdk
ddb.customVolumeMetadataName\\Hard disk 3\Volume 1 New Volume = "666666########9456:6705"
4. Power on the VM.
5. Perform vMotion.
Workaround:
Not applicable
Additional Information
To view the vmdk file, use below command.
cat disk4000.vmdk
Find the lines similar to below:
ddb.customVolumeMetadataName\\Hard disk 1\Volume 1 System Reserved = "55555###########1654:6705"
Remove the line using vi editor.
vi disk4000.vmdk
Impact/Risks:
Unable to live migrate VM to another cluster
Feedback
