vSAN witness node partitioned after an upgrade of vCenter to 7.0u3f, 7.0u3g, or 7.0U3h with Data-In-Transit Encryption enabled

Products

• VMware vSAN

Issue/Introduction

This article will assist with the remediation steps required when a customer experiences a partitioned witness with data-in-transit encryption enabled on vCenter 7.0u3f, 7.0u3g, or 7.0U3h.

Impact/Risks:

This causes stretched/2-node Stretch clusters to become partitioned.

Symptoms:

• vSAN Witness appliance is partitioned after an  upgrade to vCenter 7.0u3f, 7.0u3g, or 7.0U3h with Data in Transit Encryption enabled
• vSAN Witness Appliance is listed twice in two separate partitions in Skyline health

 

 

 

• Checking the witness unicast entries shows an empty table without the data node cluster entries

 

esxcli vsan cluster unicastagent list

NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint SubClusterUuid

----------------  --------- ---------------- ------------ ----- ---------- --------------------------------

 

• vSAN health service logs /var/log/vmware/vsan-health/vmware-vsan-health-service.log shows the following entry

2022-07-14T09:14:02.968+02:00 INFO vsan-mgmt[22229] [VsanClusterPrototypeImpl::RefreshTrustedThumbprintsOnHost opID=vsan-PC-############-W101] Host host-24765 doesn't support unicast

For details refer to KB vSAN Data in Transit encryption use of TCP port 12443

Environment

VMware vSAN 7.0.x

Cause

Known issue with vCenter 7.0.3

 

Resolution

This issue was resolved in vCenter 7.0 Update 3i (build 20845200), please patch up at the earliest convenience.

Workaround:

Action Plan only to follow on vCenter 7.0u3f, 7.0u3g or 7.0U3h.

To work around this follow the below steps:

1. Download the VsanMgmtAdapters.pyc file attached to this KB under Attachments.
2. ssh to the VCSA.
3. Move existing VsanMgmtAdapters.pyc file by running 'mv /usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanMgmtAdapters.pyc /storage/core/'
4. Upload the VsanMgmtAdapters.pyc file downloaded earlier to /usr/lib/vmware-vpx/vsan-health/pyMoVsan/ on the vCenter.
5. Run 'chmod 755 /usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanMgmtAdapters.pyc' to update the permissions of the new file
6. Restart the vsan-heath daemon
7.    vmon-cli -r vsan-health
8. ssh to the ESXi host
9. Check the witness host unicastagent list info using "esxcli vsan cluster unicastagent list"
10. After a few minutes the network partition should clear

 

Additional Information