Error: "VMotionStream timed out while waiting for disk # queue count to drop below the maximum limit of 32768 blocks" during Cross-vCenter vMotion
search cancel

Error: "VMotionStream timed out while waiting for disk # queue count to drop below the maximum limit of 32768 blocks" during Cross-vCenter vMotion

book

Article ID: 423225

calendar_today

Updated On:

Products

VMware vSphere Standard VMware vSphere ESXi

Issue/Introduction

  • Virtual machine migration between vCenters is interrupted.

  • The /var/log/hostd.log file on the source host shows the following entries:

    YYYY-MM-DD HH:MM:SS In(166) Hostd[2102001]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx] VMotionStatusCb [287422213629634110] : Prepare task completed successfully
    YYYY-MM-DD HH:MM:SS In(166) Hostd[2101966]: [Originator@6876 sub=Vcsvc.VMotion opID=#######-#######-auto-#######-##:#######-##-##-##-#### sid=##### user=vpxuser:] InitiateSource WID = #####
    YYYY-MM-DD HH:MM:SS In(166) Hostd[2101966]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx.opID=#######-#######-auto-#######-##:#######-##-##-##-####sid=###### user=vpxuser: VMotionInitiateSrc : wid=########
    YYYY-MM-DD HH:MM:SS Db(167) Hostd[2101966]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx opID=#####-#######-auto-#######-##:#######-##-##-##-####sid=###### user=vpxuser:VMotionInitiateSrc: begin ReadSynchronized
    YYYY-MM-DD HH:MM:SS Db(167) Hostd[2101966]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx opID=#####-#######-auto-#######-##:#######-##-##-##-####sid=###### user=vpxuser: VMotionInitiateSrc: end ReadSynchronized
    YYYY-MM-DD HH:MM:SS Db(167) Hostd[2101966]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx opID=#####-#######-auto-#######-##:#######-##-##-##-####sid=###### user=vpxuser: VMotionInitiateSrc: Done
    YYYY-MM-DD HH:MM:SS Db(167) Hostd[2101993]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx VMotionInitiateSrc: Start message: A fatal internal error occurred. See the virtual machine's log for more details.
    YYYY-MM-DD HH:MM:SS Db(167) Hostd[2101928]: --> VMotionStream timed out while waiting for disk 6's queue count to drop below the maximum limit of 32768 blocks. This could indicate either network or storage problems preventing proper block transfer.
    YYYY-MM-DD HH:MM:SS Db(167) Hostd[2102000]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx VMotionStatusCb [287422213629634110]: Failed with error [N3Vim5Fault20GenericVmConfigFaultE:0x000000b9f286ccd0]
    YYYY-MM-DD HH:MM:SS Db(167) Hostd[2102000]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx VMotionStatusCb: Firing ResolveCb
    YYYY-MM-DD HH:MM:SS In(166) Hostd[2102000]: [Originator@6876 sub=Vcsvc.VMotionSrc.287422213629634110] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION
    YYYY-MM-DD HH:MM:SS In(166) Hostd[2102000]: [Originator@6876 sub=Vcsvc.VMotionSrc.287422213629634110] ResolveCb: Failed with fault: (vim.fault.GenericVmConfigFault) {
    YYYY-MM-DD HH:MM:SS In(166) Hostd[2101928]: -->          message = "VMotionStream timed out while waiting for disk 6's queue count to drop below the maximum limit of 32768 blocks. This could indicate either network or storage problems preventing proper block transfer.
    YYYY-MM-DD HH:MM:SS In(166) Hostd[2101928]: --> VMotionStream [c0a80c1c:287422213629634110] timed out while waiting for disk 6's queue count to drop below the maximum limit of 32768 blocks. This could indicate either network or storage problems preventing proper block transfer.

  • The /var/log/hostd.log file on the destination ESXi host indicates connection reset errors.

    YYYY-MM-DD HH:MM:SS Db(167) Hostd[2099716]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx VMotionPrepare: MigrateFromDest message: Failed waiting for data.  Error bad004b. Connection reset by peer.
    YYYY-MM-DD HH:MM:SS Db(167) Hostd[2099724]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmx VMotionStatusCb [287422219775952989]: Failed with error [N3Vim5Fault20GenericVmConfigFaultE:0x0000008700760510]

  • The entries in /vmfs/volumes/<Datastore_Name>/<VM_Name>/vmware.log indicate timeout errors for this VM.

    YYYY-MM-DD HH:MM:SS Wa(03) vmx - Mirror: scsi0:6: Failed to copy disk: Timeout
    YYYY-MM-DD HH:MM:SS Wa(03) worker-6178497 - SVMotionMirroredModeThreadDiskCopy: Found internal error when woken up on diskCopySemaphore. Aborting storage vmotion.
    YYYY-MM-DD HH:MM:SS Wa(03) worker-6178497 - SVMotionCopyThread: disk copy failed. Canceling Storage vMotion.
    YYYY-MM-DD HH:MM:SS In(05) worker-6178497 - SVMotionCopyThread: Waiting for SVMotion Bitmap thread to complete before issuing a stun during migration failure cleanup.
    YYYY-MM-DD HH:MM:SS In(05) vmx - Migrate: Caching migration error message list:
    YYYY-MM-DD HH:MM:SS In(05) vmx - [msg.svmotion.fail.internal] A fatal internal error occurred. See the virtual machine's log for more details.
    YYYY-MM-DD HH:MM:SS In(05) vmx - [msg.svmotion.disk.copyphase.failed] Failed to copy one or more disks.
    YYYY-MM-DD HH:MM:SS In(05) vmx - [msg.mirror.disk.copyfailed] Failed to copy source (/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmdk) to destination (/vmfs/volumes/#######-######-####-#########/VM_Name/VM_Name.vmdk): Timeout.
    YYYY-MM-DD HH:MM:SS In(05) vmx - [vob.vmotion.stream.check.block.mem.timed.out] VMotionStream [c0a80c1c:287422217649189303] timed out while waiting for disk 6's queue count to drop below the maximum limit of 32768 blocks. This could indicate either network or storage problems preventing proper block transfer.
    YYYY-MM-DD HH:MM:SS In(05) vmx - Migrate: cleaning up migration state.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

This issue typically indicates a network configuration problem or a storage I/O bottleneck preventing successful block transfer during migration. Common causes include:

  • MTU Mismatch: Discrepancies in Maximum Transmission Unit (MTU) settings between source and destination ESXi hosts, or intermediate network devices (switches, routers).
  • High Latency: Round-trip time (RTT) latency exceeding 150 milliseconds.
  • Storage Bottleneck: Severe storage I/O contention causing the disk write/read queue to hit its limit.

Resolution

  1. Verify MTU Configuration: Ensure identical MTU settings on the source and destination ESXi hosts, VMkernel adapters, and all intermediate physical network infrastructure.
  2. Check Network Latency: Confirm that the network path between hosts maintains an RTT latency of 150ms or less.
  3. Investigate Storage I/O: Analyze storage performance logs for potential bottlenecks or congestion occurring during the migration phase.
  4. Engage Support Teams: If the above configurations are verified and the issue persists, involve your in-house Network or Storage administration teams to investigate further.