Unable to register VM and Power ON after storage outage

Products

• VMware vSphere ESXi

Issue/Introduction

Upon storage outage and restoring the storage, Some VMs face issue with registering and Powering ON 

Environment

vSphere 8.x

Cause

- The VM was registering but showing as "invalid" and skipping the VM, on vSphere we see the VM with full disk path.

KB 342634 (Virtual machines names appear as ds:///vmfs/volumes/datastore/vm_name) 

[root@esxi:/vmfs/volumes/ds-name/vm-name] vim-cmd solo/registervm /vmfs/volumes/ds-name/vm-name/vm-name.vmx
283
[root@esxi:/vmfs/volumes/ds-name/vm-name] vim-cmd vmsvc/getallvms 283
Skipping invalid VM '283'

- VM file locks and they were no locks. However, the issue is because the vmx file was empty this is mainly due to storage outage causing the corrupted vmx file 

 KB 313833 (VMware virtual machine file lock on VMFS datastore) 

[root@esxi:/vmfs/volumes/ds-name/vm-name] ls | while read x; do vmfsfilelockinfo -p $x| grep -i "is locked"; done
[root@LVGPEINFRA02P:/vmfs/volumes/ds-name/vm-name] cat vm-name.vmx
Register VM

- VM fails to Power ON with below after re-registering post storage outage

YYYY-MM-DDTHH:MM:SS.788Z Er(163) Hostd[17698662]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/ds-uuid/vm-name/vm-name.vmx opID=mxxxxxxw-2xxxx5-auto-6xxx-h5:7xxxxxxx1-xx-xx-xx-xx-xxxx sid=5xxxxxxx4 user=vpxuser:DOMAIN\User] Sandboxd call failed: Error:
YYYY-MM-DDTHH:MM:SS.788Z Er(163) Hostd[17698599]: -->    com.vmware.vapi.std.errors.already_exists
YYYY-MM-DDTHH:MM:SS.788Z Er(163) Hostd[17698599]: --> Messages:
YYYY-MM-DDTHH:MM:SS.788Z Er(163) Hostd[17698599]: -->    com.vmware.esx.vm_sandbox.security_domains.error<>
YYYY-MM-DDTHH:MM:SS.788Z Er(163) Hostd[17698599]: -->
YYYY-MM-DDTHH:MM:SS.791Z Db(167) Hostd[17698662]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/ds-uuid/vm-name/vm-name.vmx opID=mxxxxxxw-2xxxx5-auto-6xxx-h5:7xxxxxxx1-xx-xx-xx-xx-xxxx sid=5xxxxxxx4 user=vpxuser:DOMAIN\User] Start Vmx message:
YYYY-MM-DDTHH:MM:SS.791Z Db(167) Hostd[17698647]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/ds-uuid/vm-name/vm-name.vmx] Power On message:
YYYY-MM-DDTHH:MM:SS.791Z Db(167) Hostd[17698647]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/ds-uuid/vm-name/vm-name.vmx] Created power off monitor: security domain=; verify=true; needsPowerOn=false; task=haTask-284-vim.VirtualMachine.powerOn-00000.

Resolution

Step 1:  Restore the VM with Backup

Step 2: (Step 1 is ideal solution, Step 2 is a Workaround)

- If there are no VM backups, look for .vmx~ or .vmx.bkp (it is unlikely customers will have vmx bkp file, if it exists without VM file backups, the vmx backup will be handy)

- Copy the vmx backup into the actual vmx configuration file and registered the VM.

KB 335224 (Add or Register a Virtual Machine (VM) in vCenter Server) 

[root@LVGPEINFRA02P:/vmfs/volumes/ds-name/vm-name] cat vm-name.vmx.bkp > vm-name.vmx
[root@LVGPEINFRA02P:/vmfs/volumes/ds-name/vm-name] vim-cmd solo/registervm /vmfs/volumes/ds-name/vm-name/vm-name.vmx
284
[root@LVGPEINFRA02P:/vmfs/volumes/ds-name/vm-name] vim-cmd vmsvc/getallvms | grep 284
284    VM-NAME   [DS-NAME] vm-name/vm-name.vmx   centos64Guest                vmx-10    Panorama-9.0.1

Step 3: (Follow - up)

- If the VM fails to Power ON after successfully re-registering, vMotion the VM to a new Host and Power ON

KB 323188 (Failed to Power ON Virtual Machine)