Troubleshooting vSAN Witness appliance partitioned from the stretched cluster
search cancel

Troubleshooting vSAN Witness appliance partitioned from the stretched cluster

book

Article ID: 326958

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

  • vSAN Health displays a "Network Partition" alarm.
  • The vSAN Witness Appliance is in a partitioned/isolated state and cannot communicate with the vSAN data nodes.
  • The vSAN cluster may report objects as having reduced availability with absent components.

 



 

Environment

VMware vSAN (2-Node or Stretched Cluster)
vSAN Witness Appliance

 

Cause

This issue occurs when vSAN network communication is interrupted between the data nodes and the Witness Appliance. Common causes include:

    • Missing or incorrect static routes.

    • MTU size mismatch across the routing path.

    • vSAN traffic tagging applied to the incorrect VMkernel interface or is missing.

    • Firewalls or physical network switches blocking vSAN communication.

    • ESXi version mismatch between the witness appliance and the data nodes

Resolution

    1. Check ESXi versions 
      • Validate that the vSAN Witness Appliance and all data nodes are operating on the identical ESXi version and build. If a version mismatch exists, upgrade the Witness Appliance to match the data node version utilizing standard ESXi lifecycle management or upgrade procedures. 

      • Consult the official VMware ESXi upgrade documentation for comprehensive procedures, prerequisites, and further technical specifications.

    2. Verify network connectivity 

        1. Verify the VMkernel interface designated for vSAN witness traffic by executing the following command:
          esxcli vsan network list

          Note: When executing network tests or configurations, ensure the correct VMkernel interface is utilized. If a VMkernel interface is explicitly tagged for vSAN Witness traffic, utilize that designated interface and its assigned IP address. In the absence of a dedicated Witness tag, default to the VMkernel interface tagged for standard vSAN traffic.

        2. Execute bidirectional vmkping tests to validate network connectivity between the vSAN data nodes and the Witness Appliance:
          vmkping -I vmkX <vSAN IP for witness traffic>

          The maximum allowable Round-Trip Time (RTT) latency between the vSAN Witness Appliance and the ESXi data nodes depends on the specific cluster topology and scale. Ensure the network architecture adheres to the following thresholds:

          • Standard Stretched Clusters (1 to 10 nodes per site): RTT must remain below 200ms.

          • Large Stretched Clusters (11 to 20 nodes per site): RTT must remain below 100ms.

          • 2-Node / ROBO Clusters: RTT must remain below 500ms.

           

        3. Verify end-to-end Maximum Transmission Unit (MTU) consistency and identify potential mismatches across the network path.
          vmkping -I vmkX <vSAN IP for witness traffic> -d -s <payload minimum>
          To accurately validate network configuration, ICMP echo requests must be transmitted with a full, unfragmented frame payload to account for header overhead. For networks configured with a standard MTU of 1500, the unfragmented payload size must be  1472 bytes. For networks utilizing jumbo frames (MTU 9000), the unfragmented payload size must be 8972 bytes.

        4. Validate that vSAN traffic is properly tagged on the designated VMkernel interface of the Witness Appliance. In a default deployment, the Witness Appliance allocates vmk0 for Management traffic and vmk1 for vSAN network communication.

          • Ensure that the "Witness" traffic tag is not erroneously applied to any VMkernel adapter on the Witness Appliance itself.
          • In Witness Traffic Separation (WTS) configurations, the "Witness" traffic type tag is designated exclusively for use on data nodes. If this tag is identified on a Witness Appliance VMkernel interface, it must be removed to ensure proper network stack

        5. Verify network connectivity from the vSAN Witness Appliance to the data nodes over port 12321 by executing the following command directly on the Witness Appliance:
          tcpdump-uw -i vmkX | grep 12321
          And Execute the following command directly on the Primary and Backup nodes:
          tcpdump-uw -i vmkX | grep <witness IP/FQDN>
           

          Successful network connectivity is verified by observing bidirectional traffic—comprising both incoming requests and corresponding responses—over port 12321 on both the Primary and Backup nodes.

          Examples: 

          • Validate that the expected IP address or Fully Qualified Domain Name (FQDN) is accurately reflected within the resulting output.
          • Only the Primary Node and the Backup will be reaching out to the witness over port 12321
          • Bidirectional communication over port 12321 must be enabled across all vSAN cluster architectures. This port is strictly required by the vSAN Cluster Monitoring, Membership, and Directory Service (CMMDS) to exchange cluster heartbeats. Verify and resolve any network connectivity issues on port 12321 prior to proceeding with further operations or diagnostics.
    3. Review firewall
      • Inspect all intervening firewalls and network security filters for dropped UDP traffic between the vSAN data nodes and the Witness Appliance on ports 12321 and 2233
        In the event of a transit path disruption, existing UDP sessions may prematurely enter a "Discard" state and become stale.
        Because matching incoming traffic will continuously refresh this discard state, the disrupted sessions will not naturally time out, you must manually flush these stale UDP sessions from the firewall state table to force the creation of a new session and restore bidirectional communication.

      • This behavior is commonly observed in firewalls managing UDP sessions, particularly when an application utilizes identical ports for both source and destination traffic. To achieve a permanent resolution, reduce the default UDP session timeout threshold within the firewall configuration.

        Example output collected from firewall logs:
        #####        unknown-udp             DISCARD FLOW       10.###.###.###[12321]/Shell/17  (10.###.###.###[12321])
#####                                          10.###.###.###[12321]/Untrusted  (10.###.###.###[12321])
    4. Review Advanced parameters
      • Verify that the specified advanced settings are configured to their default value of 0 by executing the commands below:
        esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListupdates

esxcfg-advcfg -g /VSAN/DOMPauseAllCCPs

    5. Reassociate the vSAN Witness Appliance with the cluster.
      Impact/Risks: Before disabling the stretched cluster, always confirm that all other fault domains are up and accessible.
      Once this is verified and the firewall is not dropping sessions, blocking port communication or there is no firewall in use between data sites and witness, follow these steps: 

      1. Place the vSAN Witness Appliance into maintenance mode.

      2. Disable the vSAN stretched cluster configuration via the vSphere Client by navigating to the cluster object and selecting Configure > vSAN > Fault Domains and Stretched Clusters, then proceeding to disable the stretched cluster topology.

      3. For environments NOT utilizing a Shared Witness configuration, establish an SSH connection to the vSAN Witness Appliance and confirm the disk group/storage pool was removed by running vdq -iH. If the disk group is still present manually remove the disk group via the below steps.
        Impact/Risks: Prior to removing a vSAN disk group, strictly verify that you are connected to the correct ESXi host and have accurately identified the target disk group UUID or devices. It is highly recommended to place the host into maintenance mode before proceeding with the removal.

        See How to manually remove and recreate a vSAN disk group using esxcli for more details on this process

        • Execute the following command to remove the disk group on the non-shared vSAN Witness Appliance:
          For OSA enabled clusters: 
          esxcli vsan storage remove -u <VSAN Disk Group UUID>
                        or
esxcli vsan storage remove -s <VSAN Disk Group Cache Identifier>

          For ESA enabled clusters: 

          esxcli vsan storagepool remove -u <VSAN Device UUID>
                        or
esxcli vsan storagepool remove -d <VSAN Device ID>
      4. Re-enable the stretched cluster configuration by following the setup wizard, ensuring that new partitions or disks are created to host the witness components.
        • This process should successfully re-establish cluster membership and initiate the reconstruction of witness components on the newly provisioned virtual disks.
        • If the cluster fails to reform or the components do not rebuild, a redeployment of a new vSAN Witness Appliance may be required.

Additional Information

Consult the official VMware documentation below for comprehensive details regarding vSAN cluster architecture.

Refer to below documents for more details on additional troubleshooting and steps:

Powered by Wolken Software