How to extend Guest OS shutdown and force power-off timeouts in VMware HCX
Article ID: 445192
Updated On:
Products
Issue/Introduction
During VMware HCX migrations, virtual machines may require more time than the default allocation to successfully complete a Guest OS shutdown or a forced power-off. If the operation exceeds the default timeout period, the migration may fail or stall.
This article provides the steps to increase the retry count parameters, thereby extending the allowed time for these operations.
Environment
HCX 4.11.5 and later
Cause
By default, the VMware HCX polls the shutdown or power-off status every 20 seconds with a maximum of 15 retries. This results in a default hard timeout of 300 seconds (5 minutes). For larger VMs or specific OS configurations, 300 seconds may be insufficient.
Resolution
To extend the time delay for these operations, modify the retry count parameters in the HCX replication transfer service configuration and restart the application engine.
1. Backup the configuration file:
Log in to the HCX Manager appliance via SSH and create a backup of the ReplicationTransferService.zql file before making any changes: cp /opt/vmware/deploy/zookeeper/ReplicationTransferService.zql /opt/vmware/deploy/zookeeper/ReplicationTransferService.zql.bkp
2. Modify the retry parameters:
Open /opt/vmware/deploy/zookeeper/ReplicationTransferService.zql using a text editor (such as vi) and locate the following parameters:
forcePowerOffVmRetryCountvmGuestShutdownRetryCount
Increase the default value (15) to your required value. The total timeout in seconds is calculated as [Value] * 20.
Example: Changing the value to
50will result in a 1000-second timeout (50 retries × 20 seconds).
3. Restart the application engine service:
Apply the configuration changes by restarting the app-engine service:
systemctl stop app-enginesystemctl start app-engine
You can verify that the new vmGuestShutdownRetryCount has been applied by checking app.log. The retry count is logged incrementally in app.log while the Guest OS shutdown task runs.
Search /common/logs/admin/app.log for the specific Migration ID.
Sample log output:
YYYY-MM-DD HH:MM:SS.MMM UTC [ReplicationTransferService_SvcThread-XX, Ent: <Entity>, , TxId: <TxID>] INFO c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown for migration <Migration_ID>YYYY-MM-DD HH:MM:SS.MMM UTC [ReplicationTransferService_SvcThread-XX, Ent: <Entity>, , TxId: <TxID>] INFO c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 1 for migration <Migration_ID>YYYY-MM-DD HH:MM:SS.MMM UTC [ReplicationTransferService_SvcThread-XX, Ent: <Entity>, , TxId: <TxID>] INFO c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown for migration <Migration_ID>YYYY-MM-DD HH:MM:SS.MMM UTC [ReplicationTransferService_SvcThread-XX, Ent: <Entity>, , TxId: <TxID>] INFO c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 2 for migration <Migration_ID>YYYY-MM-DD HH:MM:SS.MMM UTC [ReplicationTransferService_SvcThread-XX, Ent: <Entity>, , TxId: <TxID>] INFO c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown for migration <Migration_ID>
Additional Information
- Can this change be applied to in-flight migrations?
- Yes, as long as it is done before the migration switchover starts. Note that restarting the
app-enginewill cause a brief interruption to management services.
- Yes, as long as it is done before the migration switchover starts. Note that restarting the
- Can this change be applied to migrations that are "waiting for switchover"?
- Yes, provided the switchover has not yet been triggered.
- Is this configuration persistent?
- The changes will persist across standard appliance reboots. However, they will NOT persist across HCX version upgrades. You must reapply these changes after any system upgrade.
Feedback
