HCX virtual machine migration fails with the error: 'Failed to prepare consistent image' or/and 'Creating virtual machine image on target side failed'

Products

• VMware Cloud on AWS
• VMware HCX

Issue/Introduction

This issue affects all types of HCX migrations, including vMotion, Bulk, and RAV. 

 

Cause

The cause of the issue is that the configuration file for the virtual machine is too large, Because it exceeds the currently configured maximum allowed size, the service cannot create a consistent image of the VM.

In the following example, when we check the file /common/logs/admin/app.log and filter for the VM migration ID, we observe an error indicating that the file size is too large: it is 557,696 bytes, while the current configure maximum size of 262,144 bytes.

Resolution

Please note that the changes below need to be made on both the HCX connector and the HCX cloud IX appliance.

• Log in to IX appliance via HCX CCLI.
  • SSH to HCX Manager
  • Type "ccli"
  • Type "list" to view a list of HCX Manager nodes.
  • Identify the Node ID for the HCX Manager service to which you want to connect.
  • Type "go #" where # is the node ID.
  • Type "ssh"
    • For more information on the same refer to Using Central CLI to Connect to HCX Services
• Take a backup of the "/etc/vmware/hbrsrv.xml" by running the command "cp /etc/vmware/hbrsrv.xml /etc/vmware/hbrsrv.xml.bkp"
• Open the file "/etc/vmware/hbrsrv.xml" using the text editor.
• Update the "maxConfigFileSize" value 
• "maxConfigFileSize" is marked and we need to unmark (i.e. remove <!-- and --> before and after <maxConfigFileSize>) as shown below,
  • 
    
    • Earlier:
      
      • root@HCX-servicemesh3-IX-R1 [ ~ ]# cat /etc/vmware/hbrsrv.xml | grep maxConfigFileSize
              <!-- <maxConfigFileSize>65536</maxConfigFileSize> -->
    • Now:
      • root@HCX-servicemesh3-IX-R1 [ ~ ]# cat /etc/vmware/hbrsrv.xml | grep maxConfigFileSize
              <maxConfigFileSize>1048576</maxConfigFileSize>
• Restart the HBRSRV service using this command:
  • "systemctl restart hbrsrv.service"
• Re-run the failed migration again to verify the fix.

Note: If you redeploy the Interconnect Appliance(IX) or deploy a new ServiceMesh / Interconnect Appliance, the changes will need to be reapplied.

The default maximum configuration file size (maxConfigFileSize) is 64 KB, which equals 65536 bytes. Currently, we have it set to 1 MB, which is equal to 1048576 bytes.

Additional Information

Please open a new SR with GSS support quoting this KB if the issue is not resolved.  This is general information that has worked in the past and has been validated by Engineering.