VM Host storage migration fails with "Status: Cannot complete file creation operation".

Products

• VMware vCenter Server

Issue/Introduction

VM host and storage vmotion fails with error: 

 

"Status: Cannot complete file creation operation".

 

Host: hostd.log: 

[YYYY-MM-DDTHH:MM:SS] Er(163) Hostd[2098689]: [Originator@6876 sub=Vcsvc.OCM opID=lw98imdj-124177-auto-2nte-h5:700439##-##-##-##-##f5 sid=5262d6b5 user=vpxuser:VSPHERE.LOCAL\Administrator] Failed to create directory for host log file /vmfs/volumes/########-####-########bd20/vm-folder/vm-name_3033c042f96445a99fd700376f8a9995/vm-name-c452e18d.hlog

- 

vCenter: vpxd.log 

Please see vCenter vpxd log extract below:

[YYYY-MM-DDTHH:MM:SS] info vpxd[07148] [Originator@6876 sub=VmProv opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] Local-VC Host Datastore Migrate of poweredOff VM 'vm-name' (vm-2442, ds:///vmfs/volumes/########-####-########f164/vm-folder/vm-name.vmx) on host-2087 (x.x.x.x) in pool resgroup-2089 with ds ds:///vmfs/volumes/########-####-########f164/ to host-1006 (x.x.x.x) in pool resgroup-1008 with ds ds:///vmfs/volumes/########-####-########bd20/ with migId 4495771239224424031 as Operation: Local-VC_NonDRS_ComputeandStorageColdMigration
[YYYY-MM-DDTHH:MM:SS] info vpxd[07148] [Originator@6876 sub=VmCheck opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] CompatCheck results: (vim.vm.check.Result) [
--> (vim.vm.check.Result) {
--> vm = 'vim.VirtualMachine:########-####-####-####-########7ca3:vm-2442',
--> host = 'vim.HostSystem:########-####-####-####-########7ca3:host-1006',
--> }
--> ]
[YYYY-MM-DDTHH:MM:SS] info vpxd[07148] [Originator@6876 sub=VmProv opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] Resources reserved for VM vim.VirtualMachine:vm-2442 - time taken: 0 ms
[YYYY-MM-DDTHH:MM:SS] info vpxd[07148] [Originator@6876 sub=VslmService opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] FCDs were not found on the source VM = vm-2442,skipping VSLM callback : preFcdMigrateEvent
[YYYY-MM-DDTHH:MM:SS] info vpxd[07148] [Originator@6876 sub=vmomi.soapStub[810] opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] SOAP request returned HTTP failure; <<io_obj p:0x00007f24845b8128, h:44, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-2087/vpxa>, method: createEntry; code: 500(Internal Server Error); fault: (vim.fault.CannotCreateFile) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> file = "/vmfs/volumes/########-####-########bd20/vm-folder/vm-name_3033c042f96445a99fd700376f8a9995/vm-name-c452e18d.hlog"
--> msg = "Received SOAP response fault from [<<io_obj p:0x00007f24845b8128, h:44, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-2087/vpxa>]: createEntry
--> Received SOAP response fault from [<<io_obj p:0x0000008cda3195a8, h:27, <TCP '127.0.0.1 : 56465'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: createEntry
--> Cannot complete file creation operation."
--> }
[YYYY-MM-DDTHH:MM:SS] info vpxd[07148] [Originator@6876 sub=VslmService opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] FCDs were not found on the source VM = vm-2442,skipping VSLM callback : postFcdMigrateEvent
[YYYY-MM-DDTHH:MM:SS] warning vpxd[07148] [Originator@6876 sub=pbm opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] post migrate callback is skipped - VM migration failed
[YYYY-MM-DDTHH:MM:SS] info vpxd[07148] [Originator@6876 sub=drmLogger opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] Destination not a cluster - dropping per-vm setting for vim.VirtualMachine:vm-2442 and ignoring any affinity and anti-affinity rules that vm might be a part of on source VC
[YYYY-MM-DDTHH:MM:SS] info vpxd[06619] [Originator@6876 sub=vpxLro opID=EventManagerProcessJobs-########-######2] [VpxLRO] -- BEGIN lro-2695736 -- -- ScheduledTaskLRO --
[YYYY-MM-DDTHH:MM:SS] info vpxd[06619] [Originator@6876 sub=vpxLro opID=EventManagerProcessJobs-########-######2] [VpxLRO] -- FINISH lro-2695736
[YYYY-MM-DDTHH:MM:SS] error vpxd[07148] [Originator@6876 sub=VmProv opID=lw98imdj-124177-auto-2nte-h5:70043902-91-01] Get exception while executing action vpx.vmprov.InvokeCallbacks: N3Vim5Fault16CannotCreateFile9ExceptionE(Fault cause: vim.fault.CannotCreateFile

Environment

VMware ESXi 8.x

VMware vCenter Server 8.x

 

Cause

vm "migrate.hostLog" contain datastore full path. 

vm config should not contain full datastore path instead of e.g. "vm-name-c452e18d.hlog"

example:

vCenter vpxd.log

--> (vim.option.OptionValue) {
--> key = "migrate.hostLog",
--> value = "/vmfs/volumes/########-####-########bd20/vm-folder/vm-name_3033c042f96445a99fd700376f8a9995/vm-name-c452e18d.hlog"
--> }

Resolution

Remove that advanced config from the VM and then attempt to migrate. (vCenter will add the entry back with a valid value).

edit is "migrate.hostLog" and then set the value to "", which should result in removing it.

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-resource-management-8-0.html