Un-Stretch vSAN Cluster fails during "Enter Hosts to Maintenance Mode" Action

Products

• VMware Cloud Foundation

Issue/Introduction

In order to have a successful Un-stretch vSAN Cluster Operation, the resolution must be applied.

Symptoms:
Below are the some of these symptoms that can occur :

1. From vCenter Client UI, in the Tasks of the vSAN Cluster, which unstretch operation is executed, we see a failed task which tries to enter a host in a Maintenance Mode with the following error:
"General vSAN Error"

2. If you open a SSH connection to any of the ESXi hosts in that vSAN Cluster, and execute
"esxcli vsan cluster unicastagent list"
and in the list of nodes, there is a node with IsWitness == 1. Example:
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint SubClusterUuid
------------------------------------ --------- ---------------- ---------- ----- ---------- ----------------------------------------------------------- --------------
630e1560-fa89-afc4-fdb2-0050569913fc 0 true 10.0.4.4 12321 35:C5:72:93:19:68:BB:5C:FF:03:CF:80:61:A7:06:EC:AE:12:4B:EF a21d567f-e835-4177-bd77-86f430d82f73
630e1559-b1c4-61e4-31d3-005056999815 0 true 10.0.4.6 12321 3B:1C:C4:47:0B:88:E4:58:B1:1A:2B:BE:85:F7:79:71:19:92:A9:15 a21d567f-e835-4177-bd77-86f430d82f73
630e1562-c676-99b6-891b-005056994cc3 0 true 10.0.4.5 12321 32:AB:8C:C4:0C:A8:E4:08:F9:CC:A3:60:32:16:65:9D:B8:93:D6:A0 a21d567f-e835-4177-bd77-86f430d82f73
630e155e-e1b8-e1e2-915c-00505699ff1b 0 true 10.0.4.7 12321 43:AC:EA:1B:3B:BF:FC:19:46:1C:4C:68:D0:0F:9E:93:9E:92:53:95 a21d567f-e835-4177-bd77-86f430d82f73
630e155e-e366-5be4-6df8-005056990317 0 true 10.0.4.8 12321 41:16:63:2A:B0:A8:F7:F0:C3:6F:9F:82:31:51:65:36:6E:A9:A2:B4 a21d567f-e835-4177-bd77-86f430d82f73
630e155d-5964-c70e-e51f-0050569971b9 0 true 10.0.4.17 12321 D9:2A:FB:F8:45:FC:19:24:D8:2C:0F:01:51:C4:7C:44:62:A5:90:C7 a21d567f-e835-4177-bd77-86f430d82f73
630e155e-d2a7-7b30-8ba2-00505699a5c5 0 true 10.0.4.14 12321 10:AE:2B:B2:D7:A1:E7:0D:55:D3:80:88:8C:73:49:F4:E3:26:AA:8E a21d567f-e835-4177-bd77-86f430d82f73
630e1564-5467-54f4-554c-0050569918e6 0 true 10.0.4.13 12321 13:84:EE:C8:59:62:DF:63:F2:6B:79:FD:AE:81:C9:99:B1:23:F2:34 a21d567f-e835-4177-bd77-86f430d82f73
630e155d-ed1b-935c-550f-00505699390f 0 true 10.0.4.12 12321 E2:6E:A9:87:98:59:FE:15:26:51:80:CA:F9:8E:ED:C8:1B:E6:F7:1A a21d567f-e835-4177-bd77-86f430d82f73
630e155e-d498-48b0-4f61-00505699b2fa 0 true 10.0.4.16 12321 68:75:2C:C6:7D:F8:8D:77:DB:B6:00:8C:BD:4D:46:FF:3B:85:56:33 a21d567f-e835-4177-bd77-86f430d82f73
630e155e-e308-affa-f9e8-00505699df8f 0 true 10.0.4.15 12321 09:61:68:7E:D0:1C:8D:4E:DD:C9:15:20:6C:80:07:45:A9:EE:BA:D3 a21d567f-e835-4177-bd77-86f430d82f73
630e1579-de0e-1f68-4509-00505699e6f3 1 true 10.0.4.88 12321 a21d567f-e835-4177-bd77-86f430d82f73

Environment

Vmware Cloud Foundation 4.5

Cause

The Witness ESXi host has been configured using FQDN, not IP Address.

Resolution

a. For each ESXi host in the vSAN Cluster
    1.1. Start SSH Service
    1.2. Login in as root through SSH
    1.3. Execute the following command:
        "esxcli vsan cluster unicastagent remove -a <WITNESS-HOST-IP-ADDRESS>"

b. Remove all old snapshots for each VM in that vSAN Cluster.

c. Change the configuration of the currently applied vSAN Policy:
1. From vSphere Client UI -> Menu -> Policies and Profiles -> VM Storage Profiles -> Find the currently applied vSAN Policy   
2. Click Edit                                                                                                                                                                         
3. In the vSAN Section:

    - change "Site disaster tolerance" to "None - standard cluster"                                                                                                     
d. Retry the Unstretch workflow from the SDDC Manager UI.