VM failing to power on with warning "The storage backing virtual disk has permanent device loss. You may be able to hot remove this virtual device from the virtual machine and continue after clicking Retry. Click Cancel to terminate this session."

Products

• VMware vCenter Server

Issue/Introduction

Unable to power on the virtual machine

It fails with warning : "The storage backing virtual disk vmname.vmdk has permanent device loss. You may be able to hot remove this virtual device from the virtual machine and continue after clicking Retry. Click Cancel to terminate this session."

Environment

vcenter : 8.x / 7.x

Esxi host: 8.x / 7.x

Cause

Esxi host is unable to see the datastore inventory for a specific LUN. It shows 0B.

VM files becomes inaccessible, hence vm power on state gets hung. 

 

 

Resolution

• Login to the Esxi host via CLI.
• Check the world id for the VM:   esxcli vm process list
• Kill the VM process:   esxcli vm process kill -t=soft -w=WORLDID
• Un-register the VM :  vim-cmd vmsvc/unregister <VMID>
• Re- register the vm on another Esxi host
• Power-on the VM

Additional Information

Storage can be re scanned from the affected Esxi host 

vmkfstools -v localcli storage core adapter rescan --all

 

Re-register vm

https://knowledge.broadcom.com/external/article/335224/how-to-register-or-add-a-virtual-machine.html