Fix-up operation fails during HCX OSAM migration

Products

• VMware HCX

Issue/Introduction

VMware HCX OS Assisted Migration (OSAM) may fail when the destination HCX Manager cannot establish a guest-level connection. This failure occurs at the "prepare for fixup" stage and typically prevents automatic VMware Tools installation. Common symptoms include SOAP request errors and guest operations agent contact failures.

• The HCX Manager user interface displays the following error during the migration: Failed to perform fix-up operation on migrated VM. Failed to prepare for fixup.
  
• Failed to perform fix-up operation on migrated VM.
• Failed to run SOAP request ListFilesInGuest The guest operations agent could not be contacted.
• The HCX Manager app.log located at /common/logs/admin/app.log contains the following signature:
  

  <Time Stamp> UTC [OsAssistedMigrationService_SvcThread-1599, Ent: HybridityAdmin, , TxId: TxId: ########-####-####-####-##########] WARN  c.v.v.h.s.m.jobs.WindowsFixupJob- doWaitForReadyToFixup() - Failed to run SOAP request ListFilesInGuest The guest operations agent could not be contacted. migrationId: ########-####-####-####-##########

• Review Sentinel logs on source VM for more details. In this case we saw errors indicating a failure to log in with local user "vmwuser" which is created by OSAM workflow on the destination VM using the attached installer. 

Sentinel agent log location:

Windows:

• C:\ProgramData\VMware\HCX

Linux: 

• /var/log/vmware/hcx/

Environment

VMware HCX 4.x
Hyper-V source VM (Windows Guest OS)

Cause

The Guest OS on the source VM blocks the HCX Manager's ability to log in with the temporary "vmuser" account. This is typically caused by restrictive Group Policy Objects (GPOs) preventing local user login or Antivirus/Endpoint Security software blocking the guest operation agent.

Resolution

Install VMware tools manually on the source machine prior to starting OSAM migration. 

Prerequisites:

1. A compatible version of VMware tools.
2. Download Microsoft Orca (select "MSI Tools" feature)

Steps:

1. Open the VMware tools MSI file using Orca.
2. On the left hand side, find the entry "InstallUISequence".
3. Find the entry "VM_CheckRequirements" and Right click on this entry and select "drop row".
4. For VMware tools 11.x and above you will need to find and drop the "VM_LogStart" row from the "CustomAction" table as well.
5. Save and exit Orca
6. Copy all 3 files to your Hyper-V machine:
   1. vcredist_x64.exe
   2. vcredist_x86.exe
   3. VMware Tools64.msi  -- modified
7. Install VMware tools using the modified MSI. 

NOTE:

The above method should only be performed as a last resort. Any issues that occur from force installing VMware tools on the source machine will NOT be investigated by Broadcom. 

Every effort should be made to investigate why the OS is blocking the login for the local OSAM user 'vmuser' - which in-turn blocks the installation of VMware tools on the destination. 

Additional Information

Collecting Logs for troubleshooting HCX OSAM migration failures