vMotion times out at 20% on vSAN Stretch Cluster

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Stretch cluster
• Basic Unicast connectivity check (normal ping) fails between witness and the hosts
• MTU check fails between witness and the hosts 
• Performance data collection failed
• Site latency health is in a warning state
• When attempting to migrate a VM from an existing node in the cluster to a newly added/replaced node in the cluster it times out at 20%

vmkernel log contains entries similar to:

2018-10-21T18:24:21.409Z cpu25:550443)swapobj: SwapObjDumpVmklinkResponse:1495: {ID:0x115; type:CREATE; ret:{status:bad0001; open:{lock:s/volumes/vsan:52957ad67aabae1b-3383e8ee72d94c35/d4d6225b-7afd-7899-37e9-54ab3a8a5fc4/brcsfpgdvorc001-89b23d96.vswp.3$

2018-10-21T18:24:21.409Z cpu25:550443)swapobj: SwapObjCreate:572: Error returned from vmklink call: Failure

2018-10-21T18:24:21.409Z cpu25:550443)WARNING: Swap: 3722: Failed to create swap file '/vmfs/volumes/vsan:52957ad67aabae1b-3383e8ee72d94c35/d4d6225b-7afd-7899-37e9-54ab3a8a5fc4/brcsfpgdvorc001-89b23d96.vswp.33605' : Failure

2018-10-21T18:24:21.409Z cpu25:550443)WARNING: Swap: vm 550840: 4557: Swap initialization failed. status=Failure. failed after stage=8. world=vmm0:brcsfpgdvorc001

vmware log contains entries similar to:
 

2018-10-21T18:24:21.412Z| vmx| I125: Module 'MonitorLoop' power on failed.

2018-10-21T18:24:21.412Z| vmx| I125: VMX_PowerOn: ModuleTable_PowerOn = 0

2018-10-21T18:24:21.412Z| vmx| I125: MigrateSetStateFinished: type=2 new state=12

2018-10-21T18:24:21.412Z| vmx| I125: MigrateSetState: Transitioning from state 8 to 12.

2018-10-21T18:24:21.412Z| vmx| I125: Migrate: Caching migration error message list:

2018-10-21T18:24:21.412Z| vmx| I125: [msg.migrate.resume.fail] The VM failed to resume on the destination during early power on.

Environment

VMware vSAN 6.7.x

Cause

Static route is not in the routing table of the host to the witness.

To confirm this run the below command.

esxcli network IP route ipv4 list

Resolution

Add a static route on the node to the witness appliance by running the following:

esxcli network ip route ipv4 add -g <IP of the witness> -n <gateway of the witness>