Attempting to Migrate VMs between Dell EMC VPLEX's with XCOPY results in migration failure with Permanent Device Loss (PDL) condition noted in vmkernel.log

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When attempting to migrate a VM from one Dell EMC VPLEX to another Dell EMC VPLEX with XCOPY, the operation will fail
• PDL Condition will be observed in the /var/log/vmkernel.log for the failed XCOPY command (0x83):

2021-05-24T14:37:09.283Z cpu0:2098216)WARNING: NMP: nmp_PathDetermineFailure:3532: Cmd (0x83) PDL error (0x5/0x25/0x0) - path vmhba1:C0:T5:L35 device naa.6000144000000010203550ccf81cc040 - triggering path failover
2021-05-24T14:37:09.283Z cpu0:2098216)NMP: nmp_ThrottleLogForDevice:3872: Cmd 0x83 (0x459b1dabea00, 3202355) to dev "naa.6000144000000010203550ccf81cc040" on path "vmhba1:C0:T5:L35" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0. Act:FAILOVER
2021-05-24T14:37:09.283Z cpu0:2098216)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6000144000000010203550ccf81cc040": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2021-05-24T14:37:09.283Z cpu0:2098216)WARNING: NMP: nmp_PathDetermineFailure:3532: Cmd (0x83) PDL error (0x5/0x25/0x0) - path vmhba1:C0:T5:L35 device naa.6000144000000010203550ccf81cc040 - triggering path failover
2021-05-24T14:37:09.283Z cpu0:2098216)WARNING: NMP: nmp_PathDetermineFailure:3532: Cmd (0x83) PDL error (0x5/0x25/0x0) - path vmhba1:C0:T5:L35 device naa.6000144000000010203550ccf81cc040 - triggering path failover

Environment

VMware ESXi 6.5
VMware ESXi 6.7.x
VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

This issue occurs due to the Dell EMC VPLEX returning valid sense data that satisfies the PDL condition. As a result, we do not failback to the software datamover for the copy operation and the operation will fail.

Resolution

Dell EMC is working to change the VPLEX code to provide a valid XCOPY failure condition which will allow us to failback to the software datamover.

For more details, please see Dell EMC's KB article: VPLEX: XCOPY fails when performing storage VMotion on local volumes across clusters

Workaround:
The only workaround at this time is to disable the XCOPY VAAI feature when migrating VMs between Dell EMC VPLEX's, which will use the software datamover instead. Once the copy is completed, it is recommended to enable XCOPY again.

How to disable XCOPY: Disabling Hardware Accelerated Move (XCOPY) in ESXi.