The Storage migration, virtual machine cloning, cold migration, and snapshot operations fail in VMware ESXi with the error: No such device or address

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Attempting to perform a Storage migration, virtual machine cloning, template deployment, cold migration, or snapshot operation fails with the error:
  
  No such device or address.
  
  
• The Storage migration of one disk of a server fails at 38% with this message:
  
  Set-HardDisk : <MM/DD/YYYY> <time> PM Set-HardDisk The operation for the entity
  VirtualMachine-vm-20794 failed with the following message: "A general system error
  occurred: Storage VMotion failed to copy one or more of the VM's disks. Please
  consult the VM's log for more details, looking for lines starting with
  "CBTMotion"."</time>
  
  
• You see these failures coincide with the same time frame as the UI message when reviewing the VM vmware.log:
  
  Apr 09 13:58:31.117: Worker#0| DISKLIB-VMFS :VmfsMoveDataInt : failed to move data (No such device or address:0x60009).
  Apr 09 13:58:31.118: Worker#0| DISKLIB-LIB : DiskLibCloneUsingDM: vmkernel data mover failure. No such device or address
  Apr 09 13:58:31.376: Worker#0| DISKLIB-LIB : Failed to clone : No such device or address (393225).
  Apr 09 13:58:31.377: Worker#0| CBTMotion: scsi1:0: Clone: copy operation failed: No such device or address
  Apr 09 13:58:31.377: Worker#0| CBTMotion: scsi1:0: failed to clone disk from /vmfs/volumes/########-####-########e633/TESTVM/TESTVM_1.vmdk to /vmfs/volumes/########-####-########e633/TESTVM/TESTVM1_1.vmdk.
  Apr 09 13:58:31.377: Worker#0| CBTMotion: disk clone failed, aborting Storage VMotion.
  Apr 09 13:58:31.377: Worker#0| Msg_Post: Error
  Apr 09 13:58:31.377: Worker#0| [msg.cbtmotion.clonefail] Storage vMotion clone operation failed. Cause: No such device or address
  Apr 09 13:58:31.377: Worker#0| [msg.cbtmotion.disk.clone.failed] Disk clone failed. Canceling Storage vMotion.

Environment

VMware ESX 4.0.x
VMware vSphere ESXi 5.0
VMware ESX 4.1.x
VMware ESXi 4.1.x Embedded
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 5.1

Cause

This is a common issue, caused by an underlying Storage problem, which impacts Storage migration.

Resolution

To troubleshoot and match up the same time-frame of errors with events in the vmkernel.log:

Apr 9 13:58:28 vmkernel: 109:21:24:33.849 cpu2:29896649)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f296e40, 30086809) to device "naa.6000d31000358123000000000000002e" failed on path "vmhba3:C0:T0:L115" H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x44 0x0.
Apr 9 13:58:28 vmkernel: 109:21:24:33.850 cpu2:29896649)ScsiDeviceIO: 1691: Cmd 0x28, CmdSN 0xce2ae from world 30086809 to dev "naa.6000d31000358123000000000000002e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x44 0x0.
Apr 9 13:58:30 vmkernel: 109:21:24:35.538 cpu2:16671312)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f296e40, 30086809) to device "naa.6000d31000358123000000000000002e" failed on path "vmhba3:C0:T0:L115" H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x44 0x0.
Apr 9 13:58:30 vmkernel: 109:21:24:35.538 cpu2:16671312)ScsiDeviceIO: 1691: Cmd 0x28, CmdSN 0xce2ae from world 30086809 to dev "naa.6000d31000358123000000000000002e" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x44 0x0.

This translates to:

Host Status = H:0x0 = OK
Device Status = D:0x2 = CHECK CONDITION
Plugin Status = P:0x0 = OK

Valid sense data: 0xb 0x44 0x0.

Sense Key = 0xb = ABORTED COMMAND
Additional Sense Code/ASC Qualifier = 0x44/0x0 = INTERNAL TARGET FAILURE

While this specific condition produces an error in the vmware.log (No such device or address), other array side conditions may satisfy this too.

For more information on identifying and troubleshooting these problems, see:

• Understanding SCSI device/target NMP errors/conditions in ESX/ESXi 4.x and ESXi 5.0 (1030381)
• Troubleshooting storage issues when using VMware products (2013160)

Additional Information