HCX - Bulk/RAV migration for a VM stops in Initial Base Sync with error "Current bytes transferred value 0 is not greater than the already transferred bytes value"

Products

• VMware HCX

Issue/Introduction

Bulk/RAV migration for a VM may stick in the Initial Base Sync state. The following error can be seen on the target side:

2021-05-24 10:53:38.266 UTC [ReplicationTransferService_SvcThread-110, Ent: DEFAULT, , TxId: TxId: ########-####-####-####-##########bb] WARN  c.v.h.s.r.j.ReplicationTransferMonitor- Current bytes transferred value 0 is not greater than the already transferred bytes value 361906176, hence not updating for transfer ########-####-####-####-##########1d

2021-05-24 10:53:38.323 UTC [ReplicationTransferService_SvcThread-110, Ent: DEFAULT, , TxId: TxId: ########-####-####-####-##########2b] WARN  c.v.v.h.adapters.vr.VrServerInstance- Error returned by replicaGroup,
com.vmware.vim.binding.hbr.replica.fault.HbrRuntimeFault: Error for (datastoreUUID: "vsan:8d6bc3fc632e461b-############"), (hostId: "host-16"), (pathname: "########-####-####-####-##########c1/hbrcfg.VRID-########-####-####-####-##########1d.2.vmxf.5"), (flags: retriable, pick-new-host): Class: NFC Code: 5; NFC error: No more memory is available; Code set to: Host unable to process request.; Set error flag: retriable; Set error flag: pick-new-host; Can't write to remote file/disk /vmfs/volumes/vsan:8d6bc3fc632e461b-############/########-####-####-####-##########c1/hbrcfg.VRID-########-####-####-####-##########1d.2.vmxf.5; Tried operation 4 times, giving up.; Failed to write to file (instanceKey=2) (type=vmxf) (identifier=########################.vmxf); Writing file ###########################.vmxf (offset 0) for group VRID-#######-####-#####-####-##########1d

Environment

VMware HCX

Cause

The initial base sync for a VM during bulk/RAV migrations may freeze due to the presence of "vmxf" files with a zero byte file size. Host Based Replication (HBR) is responsible for creating the config files form vmxf files at the target side. For non-zero byte vmfx files, the replication will proceed.

Resolution

1. This issue is only applicable to target environments with vSAN datastores. It does not affect VMFS datastores..
2. The zero byte dependency is only related to vmxf files. No file types (vmsd,vmx etc.) are affected
3. The vmxf file is a configuration file for teaming features. Teaming is a feature used primarily with VMware Workstation to allow administrators to logically group virtual machines for streamlined administration.
4. Check if there is a vmxf file in the VM datastore and also check if it is non-zero (or 47 bytes), during validation stage of bulk/RAV migrations.
   ssh to source ESXi host >> /vmfs/volumes/datastore/VM
   
   
   vmxf file with non-zero byte:
   
   
   vmxf file with zero byte:
   
   
   Remove the vmxf file it is if zero bytes