VM Powers Off After Storage vMotion Fails: "Failed to Establish Transport Connection", Remote Connection Failure, vMotion Error
search cancel

VM Powers Off After Storage vMotion Fails: "Failed to Establish Transport Connection", Remote Connection Failure, vMotion Error

book

Article ID: 398268

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere ESXi 8.0

Issue/Introduction

  • Virtual machines experience unexpected power-offs following a failed attempt to migrate their storage using Storage vMotion.

  • Error Observed when checking the Failed Storage vMotion Task: "vMotion migration failed to read stream keepalive: Connection closed by remote host, possibly due to timeout Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout. Remote connection failure Failed to establish transport connection".

  • This behavior stems from a critical failure during Stage 2 of the Storage vMotion process. A review of the virtual machine's log files (/vmfs/volumes/<datastore_id>/<vm_name>/vmware.log) reveals the following error messages:

    YYYY-MM-DDTHH:MM:SSZ In(05) vmx - Module 'Migrate' power on failed.
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - VMX_PowerOn: ModuleTable_PowerOn = 0
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - SVMotion_PowerOff: Not running Storage vMotion. Nothing to do
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - SVMotion_PowerOff: Invoking file powerOff callbacks.


  • Further investigation of the log files generated during the Storage vMotion operation reveals a more detailed error:

    YYYY-MM-DDTHH:MM:SSZZ In(05) worker-28822107 - SVMotion: Enter Phase 2
YYYY-MM-DDTHH:MM:SSZZ Cr(01) worker-28822107 - PANIC: NOT_REACHED bora/lib/disklib/diskLibMisc.c:589
YYYY-MM-DDTHH:MM:SSZZ Wa(03) worker-28822107 - A core file is available in "/vmfs/volumes/<datastore_id>/<vm_name>/vmx-zdump.000"
YYYY-MM-DDTHH:MM:SSZZ In(05) worker-28822107 - Msg_Post: Error
YYYY-MM-DDTHH:MM:SSZZ In(05) worker-28822107 - [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (worker-28822107)
YYYY-MM-DDTHH:MM:SSZZ In(05)+ worker-28822107 - NOT_REACHED bora/lib/disklib/diskLibMisc.c:589     
YYYY-MM-DDTHH:MM:SSZZ In(05) worker-28822107 - [msg.panic.haveLog] A log file is available in "/vmfs/volumes/<datastore_id>/<vm_name>/vmware.log".
YYYY-MM-DDTHH:MM:SSZZ In(05) worker-28822107 - ----------------------------------------
YYYY-MM-DDTHH:MM:SSZZ In(05) worker-28822107 - Exiting


  • The vmdk present inside the vm directory would contain “monolithicFlat” entry as shown below:

/vmfs/volumes/<datastore_id>/<vm_name>/

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The virtual machine's disk (VMDK) is currently configured in the "Monolithic Flat" format. This format is incompatible with VMware vSphere disk creation requirements, causing Storage vMotion tasks to fail, potentially resulting in the virtual machine being powered off.

"Monolithic Flat" disk formats are typically associated with other VMware products such as Fusion and Horizon. To ensure proper functionality within a vSphere environment, these disks must undergo a proper conversion to a vSphere-supported disk format.

Furthermore, creating snapshots of this virtual machine will also fail, as documented in the Knowledge Base article: "Creating Snapshot on virtual machine fails with error “A specified parameter was not correct: spec.deviceChange.device”."

Resolution

To resolve the issue, we must convert the disk to a VMware vSphere-supported format because "Monolithic Flat" is not compatible.

VMware vSphere products support the following VMDK disk create types: VMFS, RDM, RDMPASSTHROUGH, VMFS_RAW, and VSANSPARSE.

In the event that the clone operation fails (which is a possibility given the existing Storage vMotion issues), the recovery options are limited to the following, officially supported methods:

  1. VMware vCenter Converter: This is the primary recommendation for converting the virtual machine and its associated disks to a compatible format. The process involves using VMware vCenter Converter Standalone to migrate the VM to a vSphere-supported disk format. Detailed instructions on how to perform this conversion can be found in the VMware documentation: https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vcenter-converter/6-6/vmware-vcenter-converter-standalone-documentation/introduction-to-converter-standalone.html

    • This method allows you to specify a target vSphere environment and the desired disk format during the conversion process.

  2. Restore from Backup with Format Conversion: An alternative recovery method involves restoring the VM from a backup. This requires engaging your backup vendor and specifically requesting a restore operation that targets a VMware vSphere-supported disk creation type.

    • Supported disk creation types in VMware vSphere include: VMFS, RDM, RDMPASSTHROUGH, VMFS_RAW, and VSANSPARSE.
    • This approach necessitates coordination with the backup vendor to ensure the restore process correctly converts the disk format to one compatible with the vSphere environment.

 

Additional Information

If the suggested solutions in the "Resolution" section (options 1 and 2) prove unsuccessful, it's crucial to investigate the VM's origin and creation platform. Once you've determined this information, engage Broadcom Support for assistance, referencing this Knowledge Base article during the case creation process to expedite the resolution. Providing details about the original platform can help Broadcom diagnose underlying compatibility issues.

Powered by Wolken Software