Failed to pwoer on Horizon VDI intermittently after recompose due to "Failed to lock the file"

Products

• VMware vSphere ESXi

Issue/Introduction

Horizon VDI VM failed to power on intermittently after recompose.

According to the vmware.log, VM failed to lock its vmdk.

YYYY-MM-DDThh:mm:ss.###Z In(05) vmx - OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/######/######_##-flat.vmdk' : Failed to lock the file (262146).
YYYY-MM-DDThh:mm:ss.###Z In(05) vmx - DISKLIB-VMFS  : "/vmfs/volumes/######/######/######_##-flat.vmdk" : failed to open (Failed to lock the file): ObjLib_Open failed. Type 3

Hostd.log also showed the same.

YYYY-MM-DDThh:mm:ss.###Z Db(167) Hostd[2099549]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/######/######/######.vmx] Power On translated error to vim.fault.FileLocked

vmkernel.log didn't indicates the vmdk was locked by another host, which means the vmdk was locked by the same host.

Environment

VMware vSphere ESXi

Cause

Power On operation was executed during CBRC operation and the VM was unable to acquire the lock of its vmdk file because CBRC already acquired lock of the vmdk file.

It can be confirmed from hostd.log.

YYYY-MM-DDThh:46:30.756Z In(166) Hostd[2099538]: [Originator@6876 sub=Vimsvc.TaskManager opID=######## sid=######## user=vpxuser:VSPHERE.LOCAL\########] Task Created : haTask--vim.CbrcManager.configureDigest-########
:::
YYYY-MM-DDThh:48:03.509Z Db(167) Hostd[2099549]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/######/######/######.vmx] Power On translated error to vim.fault.FileLocked
:::
YYYY-MM-DDThh:56:20.843Z In(166) Hostd[2099532]: [Originator@6876 sub=Vimsvc.TaskManager opID==######## sid=######## user=vpxuser:VSPHERE.LOCAL\########] Task Completed : haTask--vim.CbrcManager.configureDigest-######## Status success

Resolution

There isn't any obvious resolution since power on and CBRC operation are requested by Horizon product and ESXi are just following their requests. However, there are some ways to mitigate the issue.

If CBRC operation is slower than you expected, investigate the cause of slow CBRC operation, possibly due to underlying storage performance.

Also, disabling CBRC from Horizon perspective (a.k.a. View Storage Accelerator) could be effective, but be careful for the performance degradation since disabling CBRC might increase IO workload.