Storage vMotion fails for Virtual Volumes with RDMs: "The specified device is not a valid physical disk device"

Products

• VMware vSphere ESXi

Issue/Introduction

• Attempting to migrate a Virtual Machine (VM) with RDM disk(s) attached fails.
  
  
• The following entries are captured in the /var/run/log/hostd.log file of the source ESXi host corresponding to the time of the vMotion failure:
  
  YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099101]: [Originator@6876 sub=Vcsvc.VMotionSrc.###############] ResolveCb: Failed with fault: (vim.fault.GenericVmConfigFault) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->    faultMessage = (vmodl.LocalizableMessage) [
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->       (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->          key = "msg.svmotion.fail.internal",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->          message = "A fatal internal error occurred. See the virtual machine's log for more details.",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->       },
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->       (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->          key = "msg.svmotion.disk.createphase.fail",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->          message = "Failed to create one or more destination disks.",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->       },
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->       (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->          key = "msg.svmotion.destdisk.createfail",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->          arg = (vmodl.KeyAnyValue) [
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->             (vmodl.KeyAnyValue) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->                key = "1",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->                value = "/vmfs/volumes/########-########-####-############/testVMname/testVMname_2.vmdk"
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->             },
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->             (vmodl.KeyAnyValue) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->                key = "2",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->                value = "The specified device is not a valid physical disk device"
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->             }
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->          ],
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->          message = "Storage vMotion failed to create the destination disk /vmfs/volumes/########-########-####-############/testVMname/testVMname_2.vmdk (The specified device is not a valid physical disk device).
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: --> "
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->       }
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->    ],
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->    reason = "A fatal internal error occurred. See the virtual machine's log for more details.",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: -->    msg = "A fatal internal error occurred. See the virtual machine's log for more details.
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099069]: --> Failed to create one or more destination disks.

Environment

VMware ESXi 7.0.x
VMware ESXi 8.0.x

Cause

This issue occurs because of a mismatch between the host's current VML ID and the VML ID stored in the Raw Device Mapping (RDM) LUN mapping file.

• While the VML ID seen by the host can change over time, the VML ID recorded inside the RDM mapping file remains static after creation.

• When a Virtual Machine migration is initiated, the system queries the disk information to update it. Because the host's current VML IDs no longer match the IDs in the mapping file, the system cannot locate the correct LUNs, triggering the migration error.

Resolution

Identifying VML ID Inconsistencies:

1. Verify the VML ID recognized by the ESXi host

   To retrieve the VML ID of a LUN as seen by the host, execute the following command using the LUN's NAA ID:
   

   esxcli storage core device list -d naa.#######################

   Review the output and locate the "Other UIDs" field to find the host-recognized VML ID

   Example:
   

   esxcli storage core device list -d naa.################################

   naa.################################:
   Display Name: STORAGE-VENDOR iSCSI Disk (naa.################################)
   Has Settable Display Name: true
   Size: #########
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.################################
   Vendor: STORAGE-VENDOR
   Model: STORAGE-MODEL
   Revision: ####
   SCSI Level: #
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: true
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters: VAAI_FILTER
   VAAI Status: supported
        Other UIDs: vml.020069#################################################
      Is Shared Clusterwide: true
   Is SAS: false
   Is USB: false
   Is Boot Device: false
   Device Max Queue Depth: ###
   No of outstanding IOs with competing worlds: ##
   Drive Type: unknown
   RAID Level: unknown
   Number of Physical Drives: unknown
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false

2. Verify the VML ID referenced by the virtual machine
   

   To check the VML ID recorded in the virtual machine's RDM descriptor file, navigate to the virtual machine's datastore directory and run the following vmkfstools command:

   vmkfstools -q /vmfs/volumes/DATASTORE_NAME/VM_NAME/VM_NAME.vmdk

   Example:
   
   Executing this command against a disk named testVMname_2.vmdk yields the following output:

   vmkfstools -q /vmfs/volumes/########-########-####-############/testVMname/testVMname_2.vmdk
   Disk /vmfs/volumes/########-########-####-############/testVMname/testVMname_2.vmdk is a Passthrough Raw Device Mapping
   Maps to: vml.02007e##############################################

    

3. Compare the VML IDs obtained in Step 1 and Step 2. If the values do not match, an inconsistency exists between the host and the virtual machine regarding that specific RDM.
   
   

   Host (Step 1): Other UIDs: vml.020069################################################
   VM (Step 2):   Maps to:    vml.02007e################################################
   

To correct the VML ID inconsistency:

NOTE: Always back up the VMs before making any storage level changes

1. Shut down the virtual machine where the RDM disk is incorrectly appearing.

2. Remove the RDM(s) from the virtual machine:

   1. Right-click the VM and click Edit Settings.
   2. Note the SCSI device ID that the RDM disk is using (for example, SCSI 1:5).
   3. Remove the RDM hard disk from the VM, ensuring that the Delete from Disk option is selected, and then click OK.
      Note: Unlike VMDK files, the Delete from Disk option does not delete the data on the disk, it only removes the RDM pointer files. Data on the RDM remains intact. 
      
      
3. Re-attach the RDM disk(s) to the VM in the same slot noted in step 2.
   Note: When re-attaching RDMs, ensure the Storage Policy is set to 'Datastore Default' or a compatible non-encrypted policy if the LUN does not support provider-side encryption.

The VML ID in the RDM descriptor file will now match the VML ID seen currently by the host.

Additional Information

• This issue can also arise when performing both compute and storage migration if the RDM LUN is presented to the target host with a different LUN ID.

Related information:

• Correlating VML ID with naa ID in ESXi 7.x or later
• Migrating virtual machines with Raw Device Mappings (RDMs)