VM cloning or Storage vMotion fails with errors either "Failed to copy one or more disks" or "A fatal internal error occurred"

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Storage vMotion of a virtual machine fails.
• You see the error similar to:
  
  Failed to copy one or more disks.
   
• In the /var/log/vpxa.log file on the ESXi host, you see the error:
  
  A fatal internal error occurred. See the virtual machine's log for more details. Failed to copy one or more disks.
   
• In the /vmfs/volumes/datastore/vm_name/vmware.log file on the ESXi host, you see entries similar to:
  
  YYYY-MM-DD TIME.752Z| vmx| W110: SVMotion: scsi0:0: Failed to copy disk: I/O error
  YYYY-MM-DD TIME.752Z| Worker#1| W110: SVMotionMirroredModeThreadDiskCopy: Found internal error when woken up on diskCopySemaphore. Aborting storage vmotion.
   
• In the /var/log/vmkernel.log file of the ESXi host, you see entries similar to:
  
  YYYY-MM-DD TIME.641Z cpu4:33440)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412e40448bc0, 7812129) to dev "naa.6006016095d0280076bfa8267680df11" on path "vmhba2:C0:T1:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0. Act:NONE
  YYYY-MM-DD TIME.641Z cpu4:33440)ScsiDeviceIO: 2337: Cmd(0x412e40448bc0) 0x28, CmdSN 0x5671fa from world 7812129 to dev "naa.6006016095d0280076bfa8267680df11" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0.
  
  Or
  
  <Date-Time> cpu59:12873043)ScsiDeviceIO: 2652: Cmd(0x43be4f266680) 0x28, CmdSN 0xb062f5 from world 11097888 to dev <naa ID> failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x0 0x0.
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This issue occurs when the ESXi host is unable to read the data from the datastore where the virtual machine running during storage vMotion.

Resolution

This is an expected behavior when the disk has gone bad and returning the medium error (0x3).

 

[0x3] | MEDIUM ERROR 
11/00 | UNRECOVERED READ ERROR

 

The above error codes are sent out by the storage hardware indicating disk errors. To resolve this issue, contact the storage vendor to check and correct the errors on the disk.

For better understanding of SCSI sense codes in VMware environment, please refer Interpreting SCSI sense codes in VMware ESXi

Workaround:
Note: If the storage vendor cannot correct the disk errors, the only workaround is to recreate/restore the virtual machines from the backup.