Un-Stretch vSAN Cluster fails during "Enter Hosts to Maintenance Mode"

Products

• VMware Cloud Foundation

Issue/Introduction

In order to have a successful Un-stretch vSAN Cluster Operation, the resolution must be applied.

Symptoms:
Below are the some of these symptoms that can occur :

1. From vCenter Client UI, in the Tasks of the vSAN Cluster, which unstretch operation is executed, we see a failed task which tries to enter a host in a Maintenance Mode with the following error:
   "General vSAN Error"
2. If you open a SSH connection to any of the ESXi hosts in that vSAN Cluster, and execute "esxcli vsan cluster unicastagent list", there is a node with IsWitness == 1 in the list of nodes.

Example: 

NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint SubClusterUuid
------------------------------------ --------- ---------------- ---------- ----- ---------- ----------------------------------------------------------- --------------
630e1560-fa89-####-####-########3fc 0 true 10.0.4.4 12321 35:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:EF a21d567f-e835-####-####-########f73
630e1559-b1c4-####-####-########815 0 true 10.0.4.6 12321 3B:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:15 a21d567f-e835-####-####-########f73
630e1562-c676-####-####-########cc3 0 true 10.0.4.5 12321 32:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:A0 a21d567f-e835-####-####-########f73
630e155e-e1b8-####-####-########f1b 0 true 10.0.4.7 12321 43:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:95 a21d567f-e835-####-####-########f73
630e155e-e366-####-####-########317 0 true 10.0.4.8 12321 41:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:B4 a21d567f-e835-####-####-########f73
630e155d-5964-####-####-########1b9 0 true 10.0.4.17 12321 D9:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:C7 a21d567f-e835-####-####-########f73
630e155e-d2a7-####-####-########5c5 0 true 10.0.4.14 12321 10:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:8E a21d567f-e835-####-####-########f73
630e1564-5467-####-####-########8e6 0 true 10.0.4.13 12321 13:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:34 a21d567f-e835-####-####-########f73
630e155d-ed1b-####-####-########90f 0 true 10.0.4.12 12321 E2:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:1A a21d567f-e835-####-####-########f73
630e155e-d498-####-####-########2fa 0 true 10.0.4.16 12321 68:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:33 a21d567f-e835-####-####-########f73
630e155e-e308-####-####-########f8f 0 true 10.0.4.15 12321 09:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:D3 a21d567f-e835-####-####-########f73
630e1579-de0e-####-####-########6f3 1 true 10.0.4.88 12321 a21d567f-e835-####-####-########f73

Environment

VMware Cloud Foundation 4.5

VMware Cloud Foundation 5.x

Cause

The Witness ESXi host has been configured using FQDN, not IP Address.

Resolution

1. Change the configuration of the currently applied vSAN Policy:
   1. From vSphere Client UI -> Menu -> Policies and Profiles -> VM Storage Profiles -> Find the currently applied vSAN Policy   
   2. Click Edit                                                                                                                                                                         
   3. In the vSAN Section:
   - Change "Site disaster tolerance" to "None - standard cluster" 
   
   
2. For each ESXi host in the vSAN Cluster
   1.1. Start SSH Service
   1.2. Login in as root through SSH
   1.3. Execute the following command:
   

   esxcli vsan cluster unicastagent remove -a <WITNESS-HOST-IP-ADDRESS>

3. Remove all old snapshots for each VM in that vSAN Cluster.
4. Retry the Unstretch workflow from the SDDC Manager UI.