vMotion failing with error - "Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout." . And logs report the error code "0xbad0021"
search cancel

vMotion failing with error - "Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout." . And logs report the error code "0xbad0021"

book

Article ID: 409291

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vMotion fails with error -

"Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout.
YYYY-MM-DD T HH:MM:SS.ZZZZZZ vMotion migration [######:########] failed to send buffer to remote host: Timeout. vMotion migration [#####:#######] failed writing stream completion: Timeout Migration to host '<Destination Host IP>' failed with error Timeout (195887137).
YYYY-MM-DD T HH:MM:SS.ZZZZZZ vMotion migration [######:########] failed to read stream payload from network: Connection closed by remote host, possibly due to timeout."

 

  • Hostd logs on source host show that vMotion failed due to timed out  as below -

YYYY-MM-DD T HH:MM:SS.ZZZZZZverbose hostd[2099728] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/6610d798-4b171070-b887-0025b5b5a006/testvm/testvm.vmx opID=meksavx7-11260-auto-8ot-h5:70002541-cd-01-e4-cf81] VMotionInitiateSrc: Start message: Migration to host <10.54.1.28> failed with error Timeout (0xbad0021).
--> VMotionStream [a360111:4492267816353225661] failed writing stream completion: Timeout
--> VMotionStream [a360111:4492267816353225661] failed to send buffer to remote host: Timeout.
-->
.
.
message = "Migration to host <10.54.1.28> failed with error Timeout (0xbad0021).
--> "
-->       },
-->       (vmodl.LocalizableMessage) {
-->          key = "vob.vmotion.stream.completion.write.fail",
.
.
message = "VMotionStream [a360111:4492267816353225661] failed writing stream completion: Timeout
--> "
-->       },
-->       (vmodl.LocalizableMessage) {
-->          key = "vob.vmotion.stream.channel.write.completion.send.mbuf.fail",
-->          arg = (vmodl.KeyAnyValue) [

 

 

 

  • vmkernel logs on source host show the same timed out logs and may also reports low  bandwidth( 0bytes/s) -

YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)WARNING: VMotionUtil: 9468: 4492267816353225661 S: failed to send buffer to remote host: Timeout.
YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)WARNING: VMotionUtil: 2631: 4492267816353225661 S: failed writing stream completion: Timeout
YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)WARNING: VMotionUtil: 4074: 4492267816353225661 S: Stream completion work failed: Timeout
YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)Migrate: 102: 4492267816353225661 S: MigrateState: Failed
YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu25:2367193)WARNING: Migrate: 256: 4492267816353225661 S: Failed: Timeout (0xbad0021) @0x4200203b4ebe
YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu38:2367043)WARNING: Migrate: 6463: 4492267816353225661 S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu38:2367043)Hbr: 3815: Migration end received (worldID=2367024) (migrateType=1) (event=1) (isSource=1) (sharedConfig=1)
YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu35:2367195)VMotionUtil: 7497: 4492267816353225661 S: Socket 0x431a6fc92590 streamSock pending: 0/563272 snd 0 rcv
.
.
.
YYYY-MM-DD T HH:MM:SS.ZZZZZZ cpu41:2367192)VMotion: 2199: 4492267816353225661 S: VMotion bandwidth in last 1s: 0 bytes/s, 10s: 0 bytes/s, 20s: 0 bytes/s, 60s: 0 bytes/s,

 

Environment

  • vSphere vCenter
  • vSphere ESXi

Cause

  • The error code 0xbad0021 indicate timeout.
  • These timeouts can cause due to Physical Network latency , ESXi performance / responsiveness and VM performance / responsiveness.
  • But usually it is because of Network latency only.

Resolution

  • It is recommended to check if there is any physical network latency or congestion.
  • Perform iperf test between the source and destination ESXi hosts' vMotion Network.
  • Perform packet capture between source and destination ESXi hosts' vMotion Network while vMotion is ongoing and fails , and then check if there are any TCP retransmissions or intermittent connection resets.
  • Check VMNIC stats of both source and destination ESXi host for any drops/errors.
  • Make sure that both source and destination ESXi hosts' driver/firmware are compatible and up to date as per our Hardware Compatibility Guide.
  • Check if hostd service of the destination host is crashing
  • Check if host is flapping from Not-responding / Responding state

Additional Information

Powered by Wolken Software