VM Host storage migration fails with "Status: Cannot complete file creation operation".
Article ID: 370490
Updated On:
Products
Issue/Introduction
Symptoms:
VM host and storage vmotion fails with error:
"Status: Cannot complete
Host: hostd.log:
2024-06-07T12:32:33.953Z Er(163) Hostd[2098689]: [Originator@6876 sub=Vcsvc.OCM opID=lw98imdj-124177-auto-2nte-h5:700439##-##-##-##-##f5 sid=5262d6b5 user=vpxuser:VSPHERE.LOCAL\Administrator] Failed to create directory for host log file /vmfs/volumes/########-####-########bd20/vm-folder/vm-name_3033c042f96445a99fd700376f8a9995/vm-name-c452e18d.hlog
-
Vcenter: vpxd.log
Please see vcenter vpxd log extract below:
2024-06-07T12:32:57.232Z info vpxd[07148] [Originator@6876 sub=VmProv opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] Local-VC Host Datastore Migrate of poweredOff VM 'vm-name' (vm-2442, ds:///vmfs/volumes/########-####-########f164/vm-folder/vm-name.vmx) on host-2087 (x.x.x.x) in pool resgroup-2089 with ds ds:///vmfs/volumes/########-####-########f164/ to host-1006 (x.x.x.x) in pool resgroup-1008 with ds ds:///vmfs/volumes/########-####-########bd20/ with migId 4495771239224424031 as Operation: Local-VC_NonDRS_ComputeandStorageColdMigration
2024-06-07T12:32:57.241Z info vpxd[07148] [Originator@6876 sub=VmCheck opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] CompatCheck results: (vim.vm.check.Result) [
--> (vim.vm.check.Result) {
--> vm = 'vim.VirtualMachine:########-####-####-####-########7ca3:vm-2442',
--> host = 'vim.HostSystem:########-####-####-####-########7ca3:host-1006',
--> }
--> ]
2024-06-07T12:32:57.305Z info vpxd[07148] [Originator@6876 sub=VmProv opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] Resources reserved for VM vim.VirtualMachine:vm-2442 - time taken: 0 ms
2024-06-07T12:32:57.305Z info vpxd[07148] [Originator@6876 sub=VslmService opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] FCDs were not found on the source VM = vm-2442,skipping VSLM callback : preFcdMigrateEvent
2024-06-07T12:32:57.328Z info vpxd[07148] [Originator@6876 sub=vmomi.soapStub[810] opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] SOAP request returned HTTP failure; <<io_obj p:0x00007f24845b8128, h:44, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-2087/vpxa>, method: createEntry; code: 500(Internal Server Error); fault: (vim.fault.CannotCreateFile) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> file = "/vmfs/volumes/########-####-########bd20/vm-folder/vm-name_3033c042f96445a99fd700376f8a9995/vm-name-c452e18d.hlog"
--> msg = "Received SOAP response fault from [<<io_obj p:0x00007f24845b8128, h:44, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-2087/vpxa>]: createEntry
--> Received SOAP response fault from [<<io_obj p:0x0000008cda3195a8, h:27, <TCP '127.0.0.1 : 56465'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: createEntry
--> Cannot complete file creation operation."
--> }
2024-06-07T12:32:57.330Z info vpxd[07148] [Originator@6876 sub=VslmService opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] FCDs were not found on the source VM = vm-2442,skipping VSLM callback : postFcdMigrateEvent
2024-06-07T12:32:57.330Z warning vpxd[07148] [Originator@6876 sub=pbm opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] post migrate callback is skipped - VM migration failed
2024-06-07T12:32:57.330Z info vpxd[07148] [Originator@6876 sub=drmLogger opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] Destination not a cluster - dropping per-vm setting for vim.VirtualMachine:vm-2442 and ignoring any affinity and anti-affinity rules that vm might be a part of on source VC
2024-06-07T12:32:57.331Z info vpxd[06619] [Originator@6876 sub=vpxLro opID=EventManagerProcessJobs-########-######2] [VpxLRO] -- BEGIN lro-2695736 -- -- ScheduledTaskLRO --
2024-06-07T12:32:57.331Z info vpxd[06619] [Originator@6876 sub=vpxLro opID=EventManagerProcessJobs-########-######2] [VpxLRO] -- FINISH lro-2695736
2024-06-07T12:32:57.330Z error vpxd[07148] [Originator@6876 sub=VmProv opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] Get exception while executing action vpx.vmprov.InvokeCallbacks: N3Vim5Fault16CannotCreateFile9ExceptionE(Fault cause: vim.fault.CannotCreateFile
Environment
VMware ESXi, 8.0.2
VMware VirtualCenter version=8.0.2
Cause
vm "migrate.hostLog" contain datastore full path.
vm config should not contain full datastore path instead of e.g. "vm-name-c452e18d.hlog")
example:
vcenter vpxd.log
--> (vim.option.OptionValue) {
--> key = "migrate.hostLog",
--> value = "/vmfs/volumes/########-####-########bd20/vm-folder/vm-name_3033c042f96445a99fd700376f8a9995/vm-name-c452e18d.hlog"
--> }
Resolution
Remove that advanced config from the VM and then attempt to migrate. (vCenter will add the entry back with a valid value).
edit is "migrate.hostLog" and then set the value to "", which should result in removing it.
Feedback
