Witness appliance partitioned during ESXi upgrade from 7.x to 8.x

Products

• VMware vSAN

Issue/Introduction

Symptoms : 

• The vSAN Health Service reports the following alert: "Witness host is partitioned." after upgrading from 7.x to 8.x
• One or more vSAN objects are reported as being in a state of reduced availability, with no rebuild in progress.
• vCenter fault domain view showed that the vSAN cluster has 2 witness nodes configured.
• Fault domain configuration type : Extended Cluster

Environment

vSAN 7.x

vSAN 8.x

Cause

The issue occurs when multiple vSAN Witness Appliances are deployed with identical UUIDs. This typically results from cloning an existing Witness VM or deploying multiple appliances using the same OVA/OVF image without ensuring a unique UUID is generated for each instance. 

It is important to observe that these symptoms may arise during cluster refresh operations, particularly when the system accesses the witness appliance after a partition has occurred.

 

Resolution

Resolution: 

• Re-deploy the Witness Appliance from the official OVA. Refer : Deploying a vSAN Witness Appliance & Download location of vSAN Witness Appliance and File Service Appliance

• Note :

  • Avoid cloning existing appliances, as this can lead to duplicate UUIDs, which may cause multiple stretched clusters to unintentionally share the same witness.

  • If you're using a common image across environments (e.g., in a Private Cloud setup), ensure that each cluster has a separately deployed Witness Appliance to maintain unique UUIDs.

Recommendation:

• Consider creating a unique Witness Appliance image per cluster or region to support scalability and prevent UUID conflicts across multiple vSAN stretched clusters.

Workardound: 

1. Remove Witness host configuration: Use the RVC (Ruby vSphere Console) or vSphere Client to remove both the witness host configuration from the cluster (Refer KB 326817). This temporarily reverts the cluster to a standard configuration.
2. Re-enable and Select Witness: Re-enable the stretched cluster configuration, ensuring you manually select only the correct witness host during the wizard.
3. Verify Object Health: Run localcli vsan debug object health summary get on an ESXi host to ensure objects are transitioning from "reduced availability" back to "healthy."
4. Clean Up Environment: Advise the customer to decommission the duplicate witness appliance to prevent future management conflicts.