Unable to power-on vSAN VM after inflating VMDK

Products

VMware vSphere ESXi

Issue/Introduction

Thin provisioned VMDK residing on a vSAN datastore was inflated using Inflate Thin Virtual Disks directly from a host.

VM fails to power on due to 'cannot find file' and 'disk lock' messages.

On vCenter:

In VMware log

Cannot open the disk: No File named: MyVM _-flat.vmdk in location: /vmfs/volumes/vsan:526659736d8695d0-#############/d9e65b69-####-####-#######/

Environment

vSAN (All Versions)

Cause

Inflating a VMDK from a host directly for vSAN VM's is not supported.

Inflating a VMDK directly from a host of vCenter, uses process intended for external storage VMFS formats, where a flat file exists. Where until the task completes or fails, the VMDK may be considered a flat file by vCenter and ESXi. This will cause a power on task failure, where it believes the VMDK has a flat file extension.

Resolution

To confirm this is the issue you are experiencing, do the following.

Verify the locks on the vSAN VMDKs for the VM in question.

1: Log into the host CLI/SSH where the inflate was done.

2: Navigate to the VM, using 'cd' [The full path to the VM, can be copied from 'Edit settings' under the VM, or observed from error]

cd /vmfs/volumes/vsan:344567789-#######/VM_NAME/

3: Confirm there are no flat files in the folder (vSAN does not use flat files, but these could have been uploaded into vSAN).

ls -lathr | grep -i flat.vmdk

4: Find the vSAN hidden lock files.

ls -lathr | grep -i .*.lck

5: Inspect the files for active locks (against locks in step 3).

vmfsfilelockinfo -p {Lock_File}

*If the VM is powered off, and the locks report a host is locking the file. This means there is an active process locking these files. Where it will show exclusive lock, and indicate the MAC address of that host (tied to a physical NIC). Where in most cases, fault domain manager can provide you a friendly name as well.

*Ensure you are connected in CLI/SSH to the host that owns the lock.

6: Confirm and find the open file/process.

You can use the command below against the lock file to verify the process (ensure this is ran on the host that has the lock active, reported from 'vmfsfilelockinfo').

lsof | grep -i {Lock_File}

7: Confirm process is still open by hostd and hostd workers. (using ID from Step 5)

ps | grep {Hostd_ProcessID}

Workaround:

If you are experiencing this issue by inflating a vSAN VMDK, the task can be cleared by restarting the management agents where the task was ran from. [From SSH/CLI of host]

/etc/init.d/vpxa restart
/etc/init.d/hostd restart

Prevention:

To use thick provisioned VMDKs on vSAN, this is done by setting the VM policy for vSAN to have object reservation 100%. It is recommended to create a new policy, with the setting and set for the specific VM you wish to have thick provisioning.