VM Storage vmotion task fails with error 'Could not complete network copy for file /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/test/test_2.vmdk

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Storage vmotion for VM fails with the error  'Could not complete network copy for file /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/test/test_2.vmdk'
  
  
• vmkfstools -x check test_2.vmdk indicates disk is error free
  
  [admin@esx1:/vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/test] vmkfstools -x check test_2.vmdk
  Disk is error free
  [admin@esx1:/vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/test]
• hexdump -C test_2.vmdk do not indicate any errors for the mentioned vmdk.
  
  
• We noticed the source datastore mapped to the VM had one of the devices in an offline state.
  
  vmkfstools -Ph -v10 /vmfs/volumes/TestDatastore
  
  ********** Lines Removed ********** 
          (device naa.********************123:1 might be offline)
          (One or more partitions spanned by this volume may be offline)
  ********** Lines Removed ********** 

Environment

VMware ESXi 7.X

VMware ESXi 8.X

Cause

The offline naa.********************123 device mapped to the test VM Source datastore was causing the storage vmotion issue.

The vobd.log indicates the device naa.********************123 was turned off by the administrator

2024-09-14T20:58:08.200Z: [scsiCorrelator] 7905600166125us: [vob.scsi.device.state.off] Device naa.********************123  has been turned off administratively.
2024-09-14T20:58:08.200Z: [scsiCorrelator] 7905663924523us: [esx.problem.scsi.device.state.off] Device: naa.********************123  has been turned off administratively

 

Resolution

Option 1: If the presented lun is healthy and presented to the ESXi host then we proceed with the following steps else follow Option 2. 

1. Attach the offline Disk back to the ESXi host. 

Steps to attach an offline LUN from an ESXi host using the vSphere Client.

• Login to vSphere Web Client 
• Select the Host >> click on configure tab >> Storage Devices >> Select the offline lun >> Click on Attach
  
  

Note : If the datastore is shared across the ESXi hosts in your cluster then you will have repeat the same steps for all the ESXi host. 

2. Once the device is attached the vobd.log indicates the device is back online.

2025-01-17T11:03:50.575Z: [vmfsCorrelator] 18669854808277us: [vob.vmfs.extent.online] Device naa.********************123:1 backing file system [TestDatastore, xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx] came online
2025-01-17T11:03:50.575Z: [vmfsCorrelator] 18670006299353us: [esx.problem.vmfs.extent.online] Device naa.********************123:1 backing file system [TestDatastore, xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx] came online. This extent was previously offline. All resources on this device are now available.

Option 2: If the Device is not healthy then follow the below steps. 

If the storage device is not mapped or presented to the ESXi host, or if the storage path to the storage array is lost, we need to engage the storage vendor to resolve the offline device issue. After that, follow the steps in Option 1 to reattach the offline disk to the ESXi host.