[VMC ON AWS] HCX Bulk Migration transfer rate is slower than desired via Direct Connect

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

Information on how to troubleshoot and identify when the HCX bulk migration data replication very slow

Bulk migration data churning happens at a very slow rate, causing increased time duration for migrating VMs.

Example: 
Data churn rate for the transfer '##########################################' is 47 KBps
2022-10-05 01:16:30.784 UTC [ReplicationTransferService_SvcThread-##### Ent: DEFAULT, , TxId: ################################] INFO  c.v.h.s.r.utils.ReplicationUtil- The Virtual Machine '#######' corresponding to the transfer '##############################' is churning data at the rate of 47 KBps which is within available bandwidth 25000 KBps

The bulk migration for VMs takes longer than desired.

Environment

HCX

Cause

Generally the Direct Connect link should be quicker in terms of transmission when compared to the public internet. There are a few reasons why the DX link could be performing worse than the Public Internet link though; Mismatches in MTU's on either site can cause packet fragmentation. Any L5-7 Firewalls doing Packet Inspection or Analysis can slow down data churn rates for migrations.
 

Resolution

• Download the source logs and analyze the app.log for the migration ID on the transfer speed entries for the bulk migration replication HCX : Bulk Migration operations and best practices can be referenced for Bulk Migration operations and best practices
  
• Once the above behavior is observed immediately check the Service Mesh involved in the migration and verify the communication between source and destination is through Direct Connect.
• If Direct Connect is configured, modify the Service Mesh to run over the External Network/Internet Network Profile. Re-deploy the Service Mesh appliances to allow for the IP changes to take place, then re-test the Bulk Migration data churn speeds for the exact same test VM as before.
  
  Assuming increased transfer speeds aren't seen, verify Direct Connect configuration between the OnPrem Site and the DX provider, who terminates the connection in AWS. From the VMC standpoint, ensure the MTU values for both the SDDC and OnPrem align. As well, a review of any Firewalls at the OnPrem Edge to ensure there is no packet inspection happening which would slow down the packet transmission rates. The below KB on MTU could be referenced

         HCX is not migrating or passing traffic over network extensions at expected speeds

Additional Information