Rescuing a running virtual machine with dd when datastore metadata is inaccessible

Products

• VMware
• VMware vSphere ESXi

Issue/Introduction

This article provides information on the alternative Linux command to clone vmdks without powering off a virtual machine. You may want to use this command when:

• Metadata is corrupted or lost
• Part of the LUN has been over-written, for example, by an installation of ESX to incorrect LUN

This method may present some advantages over alternatives where the virtual machine is powered on:

• It is faster than Converter as data is transferred directly to SAN instead of over network
• It is possible to restart the guest operating system if the procedure fails

vmkfstools -i is not an option where:

• The virtual machine is powered on and the vmdk is locked
• You cannot create snapshot because you cannot risk an update to metadata

VMware Converter may not be an option where:

• The guest operating system is non-Windows. For example, Linux (pre-VMware Converter 4.0), Netware or Solaris
• Insufficient diskspace on LUN for snapshots continue to grow while the guest operating system is running

Environment

VMware ESXi 3.5.x Installable
VMware vCenter Converter Standalone 4.0.x
VMware ESXi 4.0.x Embedded
VMware Converter 3.0.x
VMware ESX Server 3.5.x
VMware ESX 4.0.x
VMware ESXi 3.5.x Embedded
VMware ESXi 4.0.x Installable

Resolution

To rescue a virtual machine with the dd command when the metadata is inaccessible:

1. Click Edit Settings > Add Hard Disk.
2. Create a new hard drive of the same size on a different datastore.
3. Reboot the guest operating system.
4. Press F2 to enter the BIOS and change boot order to boot from CD.
5. From the virtual machine console toolbar, connect the live Linux boot CD ISO as a virtual CD-ROM.
6. Press F10 to save the BIOS changes and exit.
7. Ensure that the partition table exists on the original device.
8. To copy the data to the new disk, run this command:
   
   dd if=/dev/sda of=/dev/sdb bs=16M conv=sync,noerror
   
   The time to run this command depends on the data transfer rate.

Notes:

• The virtual machine must not be powered off or reset during this process.
• Test your backup method to ensure it functions as expected in your environment.