HCX - vMotion & RAV Migration fail with Error: "Failed to write header to "/var/log/vmware/journal/" after 4.4.0 upgrade

Products

• VMware HCX

Issue/Introduction

Identify and remediate a known issue with HCX vMotion/RAV migration workflow.

Symptoms:

HCX vMotion/RAV migration for a VM may fail during run time and we can see following error:

"vMotion failed. (vmodl.fault.SystemError) { faultCause = null, faultMessage = null, reason = Failed to write header to "/var/log/vmware/journal/1659417683.38: Error while writing to file. There is no space left on the device }"

Location of App Engine log:

• HCX Manager : /common/log/admin/app.log

Cause

All migration services uses IX appliance to perform the Mobility Transfer workflow, and all subsequent workflow gets failed due to high utilization of "/var/log/" directory in the IX appliance.
After HCX manager and SM/IX appliance upgrade with 4.4.0, the usage for "/var/log" partition reaches to 100% for IX appliance.
Go to HCX admin shell >> CCLI >> List >> go <IX_Appliance> >> ssh

[root@HCXManager-IX-I1] ssh
Welcome to HCX Central CLI
root@HCXManager-IX-I1 [ ~ ]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.5G     0  1.5G   0% /dev
tmpfs           1.5G     0  1.5G   0% /dev/shm
tmpfs           1.5G   24M  1.5G   2% /run
tmpfs           1.5G     0  1.5G   0% /sys/fs/cgroup
/dev/sda2       1.7G  1.3G  383M  77% /
/dev/sda4       743M  743M     0 100% /var/log >>>>>>>>>>>
/dev/sda1        92M   53M   34M  61% /boot
/dev/sda3       378M   12M  346M   4% /var/lib

Resolution

The issue is fixed in upcoming HCX 4.4.1 release.
User is advised to upgrade once it becomes GA.

Workaround:
Open a Service Request with VMware Global Support Services and include the required information to work on remediation.

Additional Information

Impact/Risks:
All Migration services including vMotion, Cold Migration, vSR Bulk & RAV will remain affected.
There will be NO impact to Network Extension services.