vSAN 7.0 stretch clusters fail to Live vMotion VMs, create a new VM, power on a VM after a cold migration to hosts either rebooted or placed into maintenance mode and then taken back out

Products

• VMware vSAN

Issue/Introduction

Symptoms:
A reboot or maintenance mode task of a host in a vSAN stretch cluster results in the VM functions of Live vMotion, creating a new VM, power on a VM after cold migration to the rebooted/Decommed host fails

• Pro-Active VM creation test fails
• vSAN Health reports all green/healthy

Environment

VMware vSAN 7.0.x

Cause

A race condition in CMMDS where the witness network interface is published/transmitted out of sync resulting in the rebooted data host(s) of the stretch cluster failing to communicate with the witness host.

Resolution

Upgrade vCenter/ESXi to fixed in version 7.0U3c or higher

Workaround:
If you can't upgrade you can perform one of the below workarounds.

1) Place the affected host into maintenance mode, then drag the affected host out of the cluster & then back in, and then take the host out of maintenance mode
2) Untag the current vmkernel port used for witness traffic, create a dummy vmkernel port and tag it for witness traffic, then untag the dummy vmkernel port and retag the original vmkernel port for witness traffic

Both these workarounds result in re-establishing communication with the witness host again.