HCX OSAM Linux Migration Stalls at 20 Percent due to Guest Resource Exhaustion - VMware HCX

Products

• VMware HCX

Issue/Introduction

When performing a Sentinel Assisted Migration (OSAM) using VMware HCX, Linux-based mobility group jobs (specifically RHEL and Oracle Linux) may consistently stall at approximately 20% progress. The migration appears "frozen" in the HCX UI without generating an explicit error message, frequently requiring a manual cancellation of the task.

Review of the /var/log/vmware/hcx/sentinel.log on the source VM reveals SocketTimeoutException and service timeouts during the transition to the base synchronization phase.

Environment

• Product: VMware HCX 9.1.0.0 (and earlier)
• Destination: VMware Cloud Foundation (VCF) 9.1
• Source Guest OS: Red Hat Enterprise Linux (RHEL) 8.x, Oracle Linux 8.x
• In-Guest Agents: Zabbix Agent, Nessus Scanner

Cause

The 20% progress mark in an OSAM migration corresponds to the beginning of the base synchronization phase. This phase is highly resource-intensive as the Sentinel agent performs block-level check summing to identify data changes.

The stall is caused by guest OS resource exhaustion (100% CPU and RAM utilization) on the source VM. This exhaustion is triggered by the high overhead of the checksum process, which is further exacerbated by contention from third-party monitoring or security agents (such as Zabbix or Nessus). These agents compete for the same CPU cycles and memory, causing the Sentinel agent to hit internal socket timeouts and stop responding to the HCX Interconnect appliances.

Resolution

To resolve this issue, ensure the source VM has sufficient resource overhead and eliminate guest-level software contention:

1. Increase Source VM Resources: Uplift the resource allocation for the impacted source VM. It is recommended to provide at least 4 to 6 vCPUs and additional RAM (depending on the disk size being synchronized) to accommodate the checksum overhead.

2. Disable Contending Agents: Temporarily stop all in-guest monitoring, logging, or security scanning services before initiating the migration.

   • Example for Zabbix:

     systemctl stop zabbix-agent

   • Example for Nessus:

     systemctl stop nessusd

3. Restart Sentinel Service: After increasing resources and stopping contending agents, restart the Sentinel agent to clear any hung threads:

   systemctl restart vmware-hcx-osam-sentinel

4. Re-attempt Migration: Initiate the migration task again via the HCX Mobility Group.

Workaround

If a resource uplift is not possible, ensure that the migration is scheduled during a period of low application activity and that all non-critical guest services are disabled to maximize available resources for the Sentinel agent.

Additional Information

• VMware HCX Ports and Protocols

• Troubleshooting HCX OSAM Migration Stalls