Cloning or Storage vMotion fails at 18% with the error: Failed to clone: Connection timed out
search cancel

Cloning or Storage vMotion fails at 18% with the error: Failed to clone: Connection timed out

book

Article ID: 309931

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • Storage vMotion of a virtual machine fails at 18% on ESX/ESXi 4.1 hosts
  • You see the error:

    Failed to clone: Connection timed out.

  • The ESX /var/log/vmkernel log or ESXi /var/log/messages log for ESXi 4.x contains entries similar to:

    Oct 5 19:15:12 vmkernel: 26:15:27:13.225 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x83 (0x41033f4e0040) to NMP device "naa.60000970000192601889533031343443" failed on physical path "vmhba2:C0:T0:L72" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5
    Oct 5 19:15:12 0x25 0x0.
    Oct 5 19:15:12 vmkernel: 26:15:27:13.226 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x83 (0x41033f6b0540) to NMP device "naa.60000970000192601889533031343443" failed on physical path "vmhba2:C0:T0:L72" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5
    Oct 5 19:15:12 0x25 0x0.
    Oct 5 19:15:12 vmkernel: 26:15:27:13.227 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x83 (0x41033f37cb40) to NMP device "naa.60000970000192601889533031343443" failed on physical path "vmhba2:C0:T0:L72" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5
    Oct 5 19:15:12 0x25 0x0.
    Oct 5 19:15:13 vmkernel: 26:15:27:14.237 cpu4:4440)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.60000970000192601889533031343443" - issuing command 0x41033f6ba240
    Oct 5 19:15:13 vmkernel: 26:15:27:14.249 cpu4:4100)WARNING: NMP: nmpCompleteRetryForPath: Retry command 0x83 (0x41033f6ba240) to NMP device "naa.60000970000192601889533031343443" failed on physical path "vmhba2:C0:T0:L72" H:0x0 D:0x2 P:0x0 Valid sen
    Oct 5 19:15:13 se data: 0x5 0x25 0x0.

  • ESX/ESXi 3.5 and 4.0 do not experience this issue and can perform clones and Storage vMotion when sharing connectivity to the same storage


Environment

VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware ESX 4.1.x

Resolution

This issue is caused by an incorrect response provided by the attached SAN storage for a SCSI command that it did not support. According to the logs, the SAN array provided the message Illegal Request (0x5) - Logical Unit Not Supported (0x25 0x0)in response to SCSI command type 0x83 (Extended Copy). The Extended Copy command is part of the vStorage APIs for Array Integration feature in VMware ESX and ESXi 4.1.

When VMware ESX or ESXi performs a Storage vMotion task, virtual machine data is moved between disks. If ESX is connected to an array that supports vStorage APIs for Array Integration (VAAI), it attempts an Extended Copy for disk clones. In other circumstances where an array that does not support VAAI, ESX perform this move via standard disk clone using host and SAN I/O.

In the above case, the Extended Copy was not supported by the SAN array. A valid response is Illegal Request - Invalid Field in Command Descriptor Block (CDB)(as opposed to Illegal Request - Logical Unit Not Supported.) It may be possible to enable VAAI support for a SAN array with additional firmware updates, engage your array vendor.

This miscommunication results in an operation time out for the disk copy on VMware ESX and a failed Storage vMotion. As a workaround, the VAAI primitives can be disabled in VMware ESX. With this configuration, ESX performs the data copy via standard disk clone using host and SAN I/O.

To disable array hardware acceleration for Cloning and Storage vMotion operations, see Disabling the VAAI functionality in ESX/ESXi (1033665).

Note: In addition, you may have to update the firmware of the array to support VAAI.



Additional Information

VMware ESX, if connected to an array that supports VAAI, allows the array to offload workloads, such as block copying and allocation. This allows for operations such as virtual machine disk clones or Storage vMotion and block zeroing/allocations to be performed without consuming additional host computing resources or inducing unnecessary storage I/O. This capability is introduced with VMware vStorage APIs for Array Integration (VAAI) for VMware ESX/ESXi 4.1.

For more information, see vStorage APIs for Array Integration FAQ (1021976).

Note: VMware ESX/ESXi attempts to perform or test VAAI capabilities (primitives) on attached storage devices associated to an array which conforms to a VAAI rule every 15 minutes. In the above scenario, there may be benign or informational errors logged every 15 minutes against all storage devices while these primitives are tested.

For additional technical information on SCSI command 0x83, see T10/99-143r1: 7.1 EXTENDED COPY command.
Note: The preceding link was correct as of June 22, 2012. If you find the link is broken, provide feedback and a VMware employee will update the link.
Frequently Asked Questions for vStorage APIs for Array Integration
Disabling the VAAI functionality in ESXi/ESX

Powered by Wolken Software