Cloning or Storage vMotion fails at 18% with the error: Failed to clone: Connection timed out
search cancel

Cloning or Storage vMotion fails at 18% with the error: Failed to clone: Connection timed out

book

Article ID: 309931

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • Storage vMotion of a virtual machine fails at 18% on vSphere ESXi 7.x.

You see the error:

Failed to clone: Connection timed out.

  • The /var/log/vmkernel log on vmware vSphere ESXi contains entries similar to:

DDMMYYYYTHH:SS:MS vmkernel: 26:15:27:13.225 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x83 (0x41033f4e0040) to NMP device "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxx443" failed on physical path "vmhba2:C0:T0:L72" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5
DDMMYYYYTHH:SS:MS 0x25 0x0.
DDMMYYYYTHH:SS:MS vmkernel: 26:15:27:13.226 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x83 (0x41033f6b0540) to NMP device "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxx443" failed on physical path "vmhba2:C0:T0:L72" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5
DDMMYYYYTHH:SS:MS 0x25 0x0.
DDMMYYYYTHH:SS:MS vmkernel: 26:15:27:13.227 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x83 (0x41033f37cb40) to NMP device "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxx443" failed on physical path "vmhba2:C0:T0:L72" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5
DDMMYYYYTHH:SS:MS 0x25 0x0.
DDMMYYYYTHH:SS:MS vmkernel: 26:15:27:14.237 cpu4:4440)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxx443" - issuing command 0x41033f6ba240
DDMMYYYYTHH:SS:MS vmkernel: 26:15:27:14.249 cpu4:4100)WARNING: NMP: nmpCompleteRetryForPath: Retry command 0x83 (0x41033f6ba240) to NMP device "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxx443" failed on physical path "vmhba2:C0:T0:L72" H:0x0 D:0x2 P:0x0 Valid sense code
DDMMYYYYTHH:SS:MS se data: 0x5 0x25 0x0.

Environment

VMware vSphere ESXi 7.x

Resolution

This issue is caused by an incorrect response provided by the attached SAN storage for a SCSI command that it did not support. According to the logs, the SAN array provided the message Illegal Request (0x5) - Logical Unit Not Supported (0x25 0x0)in response to SCSI command type 0x83 (Extended Copy). The Extended Copy command is part of the vStorage APIs for Array Integration feature in VMware ESXi 7.x.

When VMware vsphere ESXi performs a Storage vMotion task, virtual machine data is moved between disks. If host is connected to an array that supports vStorage APIs for Array Integration (VAAI), it attempts an Extended Copy for disk clones. In other circumstances where an array that does not support VAAI, ESXi perform this move via standard disk clone using host and SAN I/O.

In the above case, the Extended Copy was not supported by the SAN array. A valid response is Illegal Request - Invalid Field in Command Descriptor Block (CDB)(as opposed to Illegal Request - Logical Unit Not Supported.) It may be possible to enable VAAI support for a SAN array with additional firmware updates, engage your array vendor.

This miscommunication results in an operation time out for the disk copy on VMware ESXi and a failed Storage vMotion. As a workaround, the VAAI primitives can be disabled in VMware ESXi. With this configuration, ESXi performs the data copy via standard disk clone using host and SAN I/O.

To disable array hardware acceleration for Cloning and Storage vMotion operations, see https://knowledge.broadcom.com/external/article/318224/disabling-the-vaai-functionality-in-ESXi.html

Note: In addition, you may have to update the firmware of the array to support VAAI.



Additional Information


VMware ESXi, if connected to an array that supports VAAI, allows the array to offload workloads, such as block copying and allocation. This allows for operations such as virtual machine disk clones or Storage vMotion and block zeroing/allocations to be performed without consuming additional host computing resources or inducing unnecessary storage I/O.

For more information, see https://knowledge.broadcom.com/external/article/323111/vsphere-storage-apis-storage-awareness.html.

Note: VMware vSphere ESXi attempts to perform or test VAAI capabilities (primitives) on attached storage devices associated to an array which conforms to a VAAI rule every 15 minutes. In the above scenario, there may be benign or informational errors logged every 15 minutes against all storage devices while these primitives are tested.

For additional technical information on SCSI command 0x83, see T10/99-143r1: 7.1 EXTENDED COPY command.

Note: The preceding link was correct as of June 22, 2012. If you find the link is broken, provide feedback and a VMware employee will update the link.

https://knowledge.broadcom.com/external/article/318941/frequently-asked-questions-for-vstorage.html.

https://knowledge.broadcom.com/external/article/318224/disabling-the-vaai-functionality-in-esxi.html.

 

Powered by Wolken Software