OSAM Densync Process Delay Due to Sparse Files

Products

• VMware HCX

Issue/Introduction

During VMware HCX OS Assisted Migration (OSAM) of Linux systems, significant delays may occur that make the migration appear to freeze or stall. This is an expected behavior when processing certain types of files.

What You'll Experience

The migration process appears to pause or freeze specifically during the densync operation phase, often when processing sparse files in the Linux system. Despite this apparent stall, the migration will eventually continue and complete successfully.

Key Details

• The delay typically adds 45-65 minutes to the total migration time
• Affects sparse files that appear large in directory listings (hundreds of GB) but consume minimal disk space (kilobytes)
• The behavior is consistent across different environments and represents expected operation of the underlying technology

Environment

• HCX
• Sparse File Virtual Machines
• Most commonly observed with sparse files that appear large in directory listings but have small actual disk usage

Cause

The densync process, which handles file replication during OSAM migrations, experiences expected delays when processing sparse files. This is not a defect in the HCX product but a fundamental characteristic of the underlying rsync technology.

Technical Explanation

1. Sparse Files Structure: These files appear to have large sizes in directory listings but actually consume minimal disk space. The large "virtual" size with small actual disk usage creates a processing challenge.
2. Densync Implementation: HCX uses a modified version of rsync with the --smart-inplace option, which combines the traditional --sparse and --inplace options to properly handle sparse files.
3. Processing Mechanics: When replicating sparse files, densync must enumerate empty blocks as zeros, and these zeros are transferred over the wire. Despite compression being enabled (via the -zz option), the combination of compression and sparse file handling remains inefficient.

Resolution

This behavior is the expected operation of the underlying rsync technology. There is currently no workaround to accelerate the process.

Recommendations for Migrations

1. Plan for Extended Time: When migrating Linux systems containing large sparse files, account for additional migration time.
2. Allow Process to Complete: The migration will appear to pause during sparse file processing but will eventually continue and complete successfully if given sufficient time.
3. Pre-Migration Assessment: Identify sparse files on Linux systems before migration using:
    

   ls -lh /path/to/suspected/sparse/file      # Shows the "virtual" size
   du -sh /path/to/suspected/sparse/file      # Shows the actual disk usage

A significant difference between these values indicates a sparse file that may cause migration delays.