vMotion failing with error - "Failed waiting for data. Error 195887137. Connection closed by remote host, possibly due to timeout."
search cancel

vMotion failing with error - "Failed waiting for data. Error 195887137. Connection closed by remote host, possibly due to timeout."

book

Article ID: 409291

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vMotion fails with error:
    "Failed waiting for data. Error 195887137. Connection closed by remote host, possibly due to timeout.
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ vMotion migration [######:########] failed to send buffer to remote host: Timeout. vMotion migration [#####:#######] failed writing stream completion: Timeout Migration to host '<Destination Host IP>' failed with error Timeout (195887137).
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ vMotion migration [######:########] failed to read stream payload from network: Connection closed by remote host, possibly due to timeout."
  • Hostd logs on source host show that vMotion failed due to timed out  as below:
    YYYY-MM-DD T HH:MM:SS.ZZZZZZverbose hostd[2099728] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/uuid/<VM Name>/<VM Name>.vmx opID=XXXXXXX-XXXXX-XXXX-XXX-XX:XXXXXXXX-XX-XX-XX-XXXX] VMotionInitiateSrc: Start message: Migration to host '<Destination Host IP>' failed with error Timeout (0xbad0021).
    --> VMotionStream [######:########] failed writing stream completion: Timeout
    --> VMotionStream [######:########] failed to send buffer to remote host: Timeout.
    -->
    .
    .
    message = "Migration to host '<Destination Host IP>' failed with error Timeout (0xbad0021).
    --> "
    -->       },
    -->       (vmodl.LocalizableMessage) {
    -->          key = "vob.vmotion.stream.completion.write.fail",
    .
    .
    message = "VMotionStream [######:########] failed writing stream completion: Timeout
    --> "

  • vmkernel logs on source host show the same timed out logs and may also reports low  bandwidth( 0bytes/s):
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)WARNING: VMotionUtil: 9468: <Migration ID> S: failed to send buffer to remote host: Timeout.
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)WARNING: VMotionUtil: 2631: <Migration ID> S: failed writing stream completion: Timeout
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)WARNING: VMotionUtil: 4074: <Migration ID> S: Stream completion work failed: Timeout
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)Migrate: 102: <Migration ID> S: MigrateState: Failed
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)WARNING: Migrate: 256: <Migration ID> S: Failed: Timeout (0xbad0021) @0x4200203b4ebe
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu38:2367043)WARNING: Migrate: 6463: <Migration ID> S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu38:2367043)Hbr: 3815: Migration end received (worldID=#######) (migrateType=1) (event=1) (isSource=1) (sharedConfig=1)
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu35:2367195)VMotionUtil: 7497: <Migration ID> S: Socket 0x431a6fc92590 streamSock pending: 0/563272 snd 0 rcv
    .
    .
    .
    YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu41:2367192)VMotion: 2199: <Migration ID> S: VMotion bandwidth in last 1s: 0 bytes/s, 10s: 0 bytes/s, 20s: 0 bytes/s, 60s: 0 bytes/s

  • Pings are working between the source and destination host.

Environment

  • vSphere vCenter
  • vSphere ESXi

Cause

  • The error code 0xbad0021 indicate timeout.
  • These timeouts can cause due to Physical Network latency , ESXi performance / responsiveness and VM performance / responsiveness.
  • But usually it is because of Network latency only.

Resolution

  • It is recommended to check if there is any physical network latency or congestion.
  • Perform iperf test between the source and destination ESXi hosts' vMotion Network.
  • Perform packet capture between source and destination ESXi hosts' vMotion Network while vMotion is ongoing and fails , and then check if there are any TCP retransmissions or intermittent connection resets.
  • Check VMNIC stats of both source and destination ESXi host for any drops/errors.
  • Make sure that both source and destination ESXi hosts' driver/firmware are compatible and up to date as per our Hardware Compatibility Guide.
  • Check if hostd service of the destination host is crashing
  • Check if host is flapping from Not-responding / Responding state

Additional Information

Powered by Wolken Software