A VM registered on a clone/snapshot LUN, fails to access all of its vmdk disks
Article ID: 377257
Updated On:
Products
Issue/Introduction
- LUNs with a vmfs volumes are cloned/replicated as part of a DR activity, or snapshots of LUNS are mounted to ESXi (e.g. as part of a VM recovery process).
- When a VM on the vmfs on the cloned/snapshot volume is registered, it does not locate all of its disks.
- VM appears as Inaccessible or Invalid in vCenter inventory
- When the snapshot LUN is presented incorrectly from SAN side, On-disk errors may be seen in vmkernel log.
YYYY-MM-DDTHH:MM:SS In (182) vmkernel: cpu46:2098459) HBX: 6221: '<Volume_Name>': HB at offset 3538944 - Lost heartbeat. On-disk:
Environment
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
Cause
- VM disks which are located within the VM folder with the .vmx file, are referenced in the VM .vmx configuration by disk name only with no pathing information, e.g.
scsi0:0.fileName = "<VM1>.vmdk".
Such a disk will be locatable on the cloned LUN when mounted. - However, disks which are on different datastores are referenced in the .vmx file by a path, in which the datastore is identified by UUID, e.g.
scsi0:1.fileName = "/vmfs/volumes/<datastore_UUID1>/<VM1>/<VM1>_1.vmdk - If a the cloned/snapshot LUNs are resignatured to mount it to ESXi, the datastores will each receive a new UUID. As a result, the path to such a disk in the .vmx file will not match the path to the recovered disk on the cloned resignatured LUN and these disks will not be locatable from the .vmx file.
- Handle Loss: Since ESXi cannot "live-remap" an active process to a new UUID, the running VMs lose their connection to persistent storage. They continue to run in memory because the process hasn't been killed, but any disk I/O request fails because the backing path no longer exists.
Resolution
For vmdks which are referenced by a full path, update the paths with the UUIDs of the cloned/replicated resignatured LUNs.
For example, one way to achieve this is:
Via an SSH session to an ESXi host, navigate to the VM folder: cd /vmfs/volumes/<datastore_UUID>/<VM1>/
Take a back up of the .vmx file.
Edit the .vmx file, updating the UUIDs for any vmdks referenced with full path. Replace the UUID with the new UUID of the cloned/replicated resignatured LUNs.
1. Force Termination of "In-Memory" VMs via vCenter
Because the VMs have lost their backing storage, a "Graceful Shutdown" via the Guest OS is impossible. You must perform a Hard Stop to release the stale file handles and metadata heartbeats.
Log in to the vSphere Client.
Locate the impacted Virtual Machine in the inventory.
Right-click the VM and select Power > Shut Down Guest OS
Note: If the standard "Shutdown guest OS" task fails or hangs due to the inaccessible datastore, proceed to the next step.
If the VM remains "Running" in the UI but is non-responsive:
Right-click the VM and select Power > Power Off (this may fail).
- Right-click the VM and select Power > Force stop (this may fail).
If unsuccessful, you must use the ESXi Command Line to kill the "zombie" process:
SSH to the host running the VM.
Run: esxcli vm process list to find the World ID.
Run: esxcli vm process kill --type=force --world-id <WID>.
2. Cleanup and Re-registration
Unregister VMs: Once the processes are stopped, right-click the "Invalid" or "Inaccessible" VMs in vCenter and select Remove from Inventory.
Synchronize Storage: Perform a cluster-wide storage rescan so all hosts recognize the new UUID:
Right-click the Cluster > Storage > Rescan Storage.
Register VMs: Browse the newly resignatured datastore, right-click the .vmx file, and select Register VM.
Power On: Start the VMs to establish fresh, valid file handles against the new UUID.
Additional Information
For LUN resignature, see Troubleshooting LUNs detected as snapshot LUNs in vSphere.
Feedback
