Troubleshooting HCX Migrations Failures

Products

• VMware HCX

Issue/Introduction

When troubleshooting HCX Migrations Failures, a specific set of data must be gathered at the time of the event. This article details what documentation is required and how to gather it prior to opening a support request with Broadcom.

Environment

HCX

Resolution

HCX Migration documentation:

• Understanding VMware HCX Bulk Migration
  • Guest OS Customization with Bulk Migration
  • Guest OS Types for Guest Customization
• Understanding HCX vMotion and Cold Migration
• Understanding HCX Replication Assisted vMotion
• Understanding HCX Assisted vMotion (Direct)
• Understanding VMware HCX OS Assisted Migration
  • Considerations for OSAM Deployment
  • Supported Guest Operating Systems
  • Guest Operating System Considerations

Pertinent Log Files (for HCX migrations failures)

HCX Manager (Connector and Cloud):

• /common/logs/admin/app.log
  • The app log is main "back-end" log for all HCX activities, including for HCX migration.

HCX Interconnect (IX-I and IX-R):

• /var/log/vmware/hbrsrv.log
  • This log is the vSphere Replication based log on the HCX Interconnect.  That explicitly means that it's exceptionally useful for reviewing each form of migration that uses vSphere Replication - Bulk Migration and Replication Assisted vMotion.  This can also be used for tracking HCX Disaster Recovery as well as the Interconnect's requirement.
• /var/log/vmware/mobility-agent.log
  • This log on the HCX Interconnect tracks primarily non replication based migrations.  In other words, both cold and vMotion migration.  This log is specifically useful to identifying VM cutover issues for those two types of migration.

ESXi host (Source and Destination):

• /var/log/vpxa.log
  • Contains information about the agent that communicates with vCenter Server
• /var/log/vmkernel.log
  • Records activities related to virtual machines and ESXi.
• /var/log/hostd.log
  • Contains information about the agent that manages and configures the ESXi host and its virtual machines.
• /vmfs/volumes/<datastore>/<vmname>/vmware.log
  • Contains virtual machine power events, system failure information, tools status and activity, time sync, virtual hardware changes, vMotion migrations, machine clones, and so on.

vCenter (Source and Destination):

• /var/log/vmware/vpxd/vpxd.log
  • The main vCenter Server log, consisting of all vSphere Client and WebServices connections, internal tasks and events, and communication with the vCenter Server Agent (vpxa) on managed ESXi  hosts.

Known issues with HCX migrations:

• HCX Bulk Migration not progressing beyond 0%
• HCX - Migration switchover failed with the error "Relocate task failed"
• HCX - OSAM Densync Process Delay Due to Sparse Files
• HCX Migration gets stuck or fails on "Collecting source details" Event step
• HCX Mobility Agent virtual host showing outdated Hypervisor version within vCenter inventory
• HCX - Migration failed with the error "Error logging into VC endpoint"
• HCX 4.8 - HCX: Bulk Migration failure with Error "Storage not accessible error observed"
• HCX - vMotion/Cold/RAV Migration unsupported for VMs having IOMMU enabled

Recommendations and Best Practices:

• HCX - Health Check and Best Practices
• HCX : Bulk Migration operations and best practices

 

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

• Error Message:
• Migration ID:
• VM Name:
• Timestamps / Time Zone:
• Service Mesh:

Handling Log Bundles for offline review with Broadcom support

• HCX Connector and HCX Cloud (Select "Collect Core HCX logs", "Collect Database Dump" and affected Service Mesh IX appliance):
  • Gather Technical Support Logs
• ESXi host (source) where the virtual machine is currently running.
  • Collecting diagnostic information for VMware ESXi