Migrating virtual machines with Raw Device Mappings (RDMs)

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article discusses some of the common questions that arise when migrating virtual machines that use Raw Device Mappings (RDMs).

Migrating virtual machines with RDMs can be performed in three ways:

• Warm migration (vMotion), with the virtual machine powered on.
• Cold migration, with the virtual machine powered off.
• Storage migration (Storage vMotion), with the virtual machine powered on.

Symptoms:

When migrating virtual machines that have disks configured with Raw Device Mappings (RDMs), you experience this symptom:
The storage vMotion fails.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Resolution

Warning:

When an RDM is removed from the virtual machine configuration, the device associated with this RDM becomes visible in the list of available devices during an attempt to increase the size of a VMFS datastore. This device can be selected during the process, which can overwrite the existing data present on that device.

To prevent such accidental lost, use one of these options:

• Inform all necessary parties involved to ensure these devices does not get selected when performing VMFS datastore size increase.
• Temporarily detach the device that has issue from all the ESXi hosts on the cluster.
  
  Note: An HBA rescan can potentially reattach these devices.

vMotion

• Files for a virtual machine are not relocated when it is migrated using vMotion.
• The virtual machine is re-registered to the destination host.
• Any RDMs remain as RDMs when the virtual machine is registered to another host. That is, no changes to the virtual machine itself are made.

Cold migration

With file relocation:

• Any non-RDM virtual disks are physically moved to the destination.
• The virtual machine configuration files are physically moved to the destination.
• Raw LUNs themselves cannot be moved, as they are raw disks presented from the SAN. However, the pointer files (RDMs) can be relocated if required.
• When performing a cold migration of a virtual machine with RDMs attached to it, the contents of the raw LUN mapped by the RDM are copied into a new .vmdk file at the destination, effectively converting or cloning a raw LUN into a virtual disk. This also applies when the virtual machine is not moving between ESXi hosts. In this process, your original raw LUN is left intact. However, the virtual machine no longer reads or writes to it. Instead, the newly-created virtual disk is used.
• If you want to cold migrate a virtual machine without cloning or converting its RDMs, remove them from the configuration of the virtual machine before migrating. You can delete the RDM from the disk when removing it (the raw LUN contents are not changed). Re-add them to the configuration when completed.
• For ESXi 5.x and later: During the migration you can use the Advanced section of the migration wizard and select if you want to maintain the same format of the files at the destination or convert it to a thick/thin disk.
• The virtual machine registration changes, but the files are left untouched.

Note: If the RDM is shared between VMs as part of clustering solutions such as MSCS or ORAC, we need to remove the RDM disk from all the other nodes and retain it with the VM where migration task will be initiated.

Cloning

If you want to clone a virtual machine without its RDMs, remove them from the configuration of the virtual machine before migrating. You can delete the RDM from the disk when removing it (the raw LUN contents are not changed, only the RDM mapping file is deleted). Re-add them to the configuration when completed.

Storage vMotion

• When you perform Storage vMotion, the virtual machine files are physically relocated to a destination datastore. The same host retains ownership or registration of the virtual machine after Storage vMotion completes.

Notes:

• In ESXi 6.x and later, during the Storage vMotion operation, virtual mode RDM data can be migrated to new VMDK disks by selecting either thick or thin provisioning for the disk type under the Advanced section of the migration wizard.
• When migrating Physical mode RDM pointer files that are greater than 2TB using Storage vMotion, you must use the vSphere Web Client.
• Physical mode RDM data can be migrated to VMDK disks only through cold migration. 
• If you do not change the destination format to thick or thin provisioned for an RDM in the advanced section of the migration wizard, only the RDM mapping file will be migrated to the destination datastore.
• If using N-Port ID Virtualization (NPIV), Storage vMotion is not supported. For more information, see the NPIV Capabilities and Limitations section in the vSphere Storage Guide .
• If you are attempting a Storage vMotion of a virtual mode RDM using the Advanced method, and the RDM pointer mapping file is already present in the target datastore, the Storage vMotion finishes quickly, but without moving the data. This is because the Storage vMotion detects that the source and target datastores for the mapping file are the same, and therefore concludes that no movement is needed.
  
  To work around this issue:
   
  1. Remove the RDM from the virtual machine, and delete it from disk to ensure that the mapping file is deleted. This will not delete the data in the RDM.
  2. Add the RDM again, and put the mapping file in a different datastore than the ultimate target of the Storage vMotion.
  3. After the virtual machine reconfiguration is complete, attempt the Storage vMotion again.
     
     Note: To avoid downtime, you can migrate the virtual mode RDM pointer file to an alternate datastore (that is, any other datastore than the actual destination datastore), then Storage vMotion the same pointer file back using either Thick Zeroed or Thick Eager Zeroed disk format. Using the du -ah command, you can confirm if the pointer exists in the directory on the alternate datastore.

For more information on the various steps involved during the migration, see Migrating virtual machines .

For instructions on using Storage vMotion, see Migration with Storage vMotion.

Notes:

• Introduction to Raw Device Mapping in vSphere
• Storage vMotion of virtual machines using virtual compatibility mode RDMs is supported. However, clustered applications or clustered virtual machine configurations do not support Storage vMotion. RDM Considerations and Limitations.
• RDM .vmdk pointer files must reside on VMFS datastores. The NFS protocol does not support SCSI commands. For more information, see Unable to add a physical mode disk mapping (RDM) to a virtual machine stored on an NFS datastore.

For Storage vMotion requirements and limitations, see the Storage vMotion Requirements and Limitations section of the vSphere Virtual Machine Administration.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
Raw Device Mapping for local storage

Snapshots

Ensure that snapshots are committed before performing storage migrations, especially if you are removing mappings or disks from your virtual machine configuration and later re-adding them. During re-addition of the respective disk, the VMware Infrastructure or vSphere Client does not allow you to re-select a specific (the previous) snapshot level or .vmdk file to add back to the virtual machine configuration.

Block size and capacity limitations of your destination datastore

Raw Device Mappings occasionally represent large raw LUNs. If you clone a virtual machine with an RDM still attached, the contents of the raw LUN mapped by the RDM are copied into a new virtual disk (.vmdk) file at the destination. Converting an RDM into a virtual disk (.vmdk) file does not succeed if your destination or target datastore does not have the capability of storing single contiguous files as large as the raw LUN. Large virtual disks have the same requirements or constraints.