VCF Greenfield Validation Stalls at 70% Due to vSAN Witness Network Partition

Products

• VMware vSAN

Issue/Introduction

During a greenfield VMware Cloud Foundation (VCF) deployment, the automated deployment or validation process stalls or becomes stuck indefinitely at 70% progress. The deployment logs or environment checks indicate a vSAN network partition affecting the vSAN Witness Host.

When analyzing the environment directly on the physical ESXi hosts, the following symptoms are observed:

• Physical ESXi hosts are completely unable to establish network connectivity to the vSAN Witness Host over the designated vSAN tagged interface, resulting in 100% packet loss when running vmkping.

• Evaluating cluster status via esxcli vsan cluster get shows an incomplete sub-cluster member count (e.g., matching only the local physical hosts and missing the Witness node metadata), confirming the cluster is partitioned.

• Network validation using vmkping from the Witness Management interface  to the witness site gateway succeeds normally.

Environment

VMware Cloud Foundation 9.1.0.0
VMware vSAN 9.1 ESA or OSA 

Cause

The upstream physical network infrastructure is misconfigured, preventing vSAN data traffic routing or VLAN traversal between the main physical cluster subnet and the isolated vSAN Witness host's dedicated vSAN VMkernel interface.

Resolution

To resolve the network partition and unblock the VCF validation workflow, configure the cluster to route vSAN storage communication explicitly across the functional witness management network layer:

1. Log in to the command-line interface (CLI) of an impacted physical ESXi host via SSH using root credentials.

2. Identify the active VMkernel interfaces and verify their assigned vSAN traffic types by running the following command:

   esxcli vsan network list

3. Test network layer connectivity to the Witness appliance over the dedicated vSAN data interface using an explicit packet size check

   vmkping -I vmkX -d -s 1472 <Witness_IP>

4. Verify whether the witness management network path can reach its local gateway by executing:

   vmkping -I vmkX -d -s 1472 <Witness_Gateway_IP>

5. If the dedicated vSAN interface (vmk2) experiences 100% packet loss but the management path  successfully communicates with the witness network path, modify the vSAN cluster configuration via the vSphere Client or SDDC Manager UI to route vSAN communication across the functional witness management network.

6. Verify successful cluster formation and node synchronization by executing the following command on the physical host:

   esxcli vsan cluster get

7. Confirm that the Sub-Cluster Member Count has updated to reflect the full inventory (all physical nodes plus the newly added Witness node) and that the Witness hostname is properly populated in the member list.

8. Resume or rerun the automated VCF validation test within the SDDC Manager interface to successfully complete the deployment past the 70% mark.

If this issue still persists contact support.

Additional Information

Intermittent vSAN cluster partitions and Object Health alarms due to Witness Traffic Separation (WTS) routing conflicts

vSAN Cluster partition

Troubleshooting vSAN Witness appliance partitioned from the stretched cluster