vSAN Health Service - Witness host not found

Products

VMware vSAN

Issue/Introduction

The Witness host not found check in the vSAN Health Service verifies that the witness host is configured and that it can be reached over the network.

If this health check is not green (OK), the vSAN stretched cluster has configuration errors that affect its operation.

Environment

VMware vSAN 7.0.x
VMware vSAN 6.7.x
VMware vSAN 8.0.x
VMware vSAN 6.6.x

Resolution

To resolve any network problems:

Configure the unicast communication between the data site host and the witness host so that the IP address and port are configured consistently on all hosts.
- To verify the listening IP address and port used by the witness host, run the esxcli vsan network list command. To avoid confusion, use only one interface configured for vSAN traffic. The witness host uses the first available vSAN network interface to listen for unicast IP traffic.
- To configure the listening address of the witness host on a host in a data site, use the esxcli vsan cluster unicastagent command. Currently, only one unicast agent address is supported.
Configure the unicast agent with the IP address and port of the witness host. Every host in the data sites uses its unicast agent to communicate with the witness host.
- Configuring vSAN Unicast networking from the command line
- Note: Unicastagent list does not need to be configured on the witness host if running ESXi version 7.0 or lower. As of version 7.0U1 and higher the unicast agent list will need to be created on the witness host.
Configure all hosts in each data site with the same IP address and port for the unicast agent. All hosts in a stretched cluster must have a consistent configuration.
Verify connectivity across the network.
- Confirm that port 12321 is open.
- Verify if there may be any firewalls or recent network changes.

If the witness host is out of service and cannot run, replace the witness host. Note that vSAN objects with FTT=1 will not be compliant until the new witness host is running and synchronized with the data sites.

Additional Information

To view the list of ESXi hosts, run the esxcli vsan cluster get command. If you run the command on the witness host, the local node type appears as WITNESS.

For example:

[root@:~] esxcli vsan cluster get Cluster Information Enabled: true Current Local Time: 2015-06-25T23:52:26Z Local Node UUID: 558c3ca0-e92f-c784-8b00-020001ff8d0e Local Node Type: WITNESS Local Node State: AGENT Local Node Health State: HEALTHY Sub-Cluster Master UUID: 558c3ca0-0439-2d57-8994-0200011a27e7 Sub-Cluster Backup UUID: 558c3ca0-1a32-1527-2847-020001469e57 Sub-Cluster UUID: 526a9252-9d60-5893-d523-c3cbeb61e1bb Sub-Cluster Membership Entry Revision: 2 Sub-Cluster Member Count: 3 Sub-Cluster Member UUIDs: 558c3ca0-0439-2d57-8994-0200011a27e7, 558c3ca0-e92f-c784-8b00-020001ff8d0e, 558c3ca0-1a32-1527-2847-020001469e57 Sub-Cluster Membership UUID: 90418c55-1ba1-5615-44ac-0200011a27e7

You can collect the list of node information and preferred fault domain by querying CMMDS from a host.

For example:

[root@:~] cmmds-tool find -t NODE owner=00000000-0000-0000-0000-000000000000(Health: Healthy) uuid=558c3ca0-1a32-1527-2847-020001469e57 type=NODE rev=0 minHostVer=0 [content = (i2 i1 i0 i1 i0)], errorStr=(null) owner=00000000-0000-0000-0000-000000000000(Health: Healthy) uuid=558c3ca0-0439-2d57-8994-0200011a27e7 type=NODE rev=0 minHostVer=0 [content = (i2 i1 i0 i1 i0)], errorStr=(null) owner=00000000-0000-0000-0000-000000000000(Health: Healthy) uuid=558c3ca0-e92f-c784-8b00-020001ff8d0e type=NODE rev=0 minHostVer=0 [content = (i2 i1 i3 i1 i0)], errorStr=(null)