Troubleshooting vSAN Encryption

Products

VMware vSAN 8.x VMware vSAN

Issue/Introduction

This article helps you troubleshoot vSAN data-in-transit encryption (DiT). It covers the following symptoms:

One or more ESXi hosts partition from the vSAN cluster. A host may rejoin and then partition again straight away.
Objects become inaccessible or show reduced availability, and VMs become inaccessible.
vSAN Health reports Data-in-transit encryption - Configuration check in warning or error, often together with vSAN cluster partition.
The Configuration check alert on a host reads “Data-in-transit encryption is disabled on the vSAN cluster but is enabled on this host”, or the reverse.
Enabling DiT on the cluster fails with “Host(s) don’t have data-in-transit encryption license”.
The affected host can still vmkping the other hosts on the vSAN network, and its unicast agent list is complete.

How DiT works:

DiT encrypts all vSAN data and metadata traffic between hosts. This includes traffic to the witness host and vSAN file services traffic. It uses AES-256 and enforces forward secrecy.
DiT does not use a key provider. Hosts generate symmetric keys themselves and exchange them when they connect to each other. A KMS or Native Key Provider (NKP) outage does not affect DiT.
DiT is independent of data-at-rest encryption. You can enable either one, or both.
DiT is a cluster-wide setting. Every host in the cluster must have the same DiT state and the same rekey interval. The default rekey interval is 1440 minutes (1 day). The range is 30 minutes to 7 days.
DiT is available from vSAN 7.0 U1 and is supported on OSA and ESA.

Environment

vSAN 8.x

vSAN 9.x

Cause

A host’s DiT setting does not match the cluster’s DiT setting.

If a host encrypts its vSAN traffic and its peers don’t, or the other way round, the peers can’t process each other’s traffic. The cluster leader rejects the mismatched host’s membership, so the host partitions. The host is still reachable on the network, so ping tests pass.

Common reasons for the mismatch:

esxcli vsan network security set was run directly on a host.
A host was reinstalled, reimaged, patched or updated, or was moved from another cluster, and came back with a different DiT setting.
DiT was used during an evaluation period and the license no longer includes it. The cluster can then no longer enable or disable DiT cleanly, and the hosts are left out of sync (see KB 373324).
A host was disconnected from vCenter when DiT was changed at cluster level, so it never received the change.
In VCF 9.1 and later, for a client cluster mounting a vSAN storage cluster’s datastore: the client and server clusters disagree on DiT for the remote datastore.

A rekey interval that differs from the cluster setting, or a DiT state that is not settled on a host, is also flagged by the Configuration check.

Resolution

Note: Don’t reboot hosts, remove them from the cluster, or put them in maintenance mode to clear the partition. A DiT mismatch survives a reboot, and evacuating data from a partitioned cluster can make objects less available. Align the DiT setting instead.
If vCenter runs on the affected vSAN datastore and is now inaccessible, use the esxcli steps below on the hosts.

1. Confirm the partition

In the vSphere Client, go to cluster › Monitor › vSAN › Skyline Health (vSAN Health in 9.x) › RETEST. Note the hosts listed under vSAN cluster partition and Data-in-transit encryption - Configuration check, and the text in the Recommendation column.
On each host, check the cluster membership:

esxcli vsan cluster get

Sub-Cluster Member Count below the number of hosts in the cluster means the cluster is partitioned. Hosts that report the same Sub-Cluster Master UUID are in the same partition.

2. Rule out membership and network causes

On each partitioned host, check the unicast agent list:

esxcli vsan cluster unicastagent list

Every other host in the cluster should be listed with the correct vSAN IP address. If an entry is missing, the cause is membership configuration, not DiT.

Check the vSAN network path, using the host’s vSAN VMkernel interface (from esxcli vsan network list):

vmkping -I vmkX -d -s 8972 <peer vSAN IP>

Use -s 1472 on a 1500 MTU network. If this succeeds, the network and MTU are not the cause.

Check that the host accepts vCenter’s membership updates:

esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates

The value should be "0".

3. Compare the DiT setting on every host

In the vSphere Client, go to cluster › Configure › vSAN › Services and check the Data-In-Transit Encryption state and rekey interval. This is the intended setting.
On every host, check its own setting:

esxcli vsan network security get

The output includes Data-in-Transit Encryption status: true or false. Any host whose value differs from the cluster setting is the cause of the partition.

Optionally, view the stored vSAN configuration on the host, including the data-in-transit section:

configstorecli config current get -c vsan -g system -k vsan

If vCenter is unavailable, the intended setting is usually the one held by the majority partition.

4. Find out why the setting changed

Check whether DiT was changed from the ESXi shell, and when:

grep -i "network security" /var/run/log/shell.log

In the vSphere Client, check cluster › Monitor › Tasks and Events for a DiT reconfiguration, and check the host’s history: was it reinstalled, patched or moved from another cluster?
Check the cluster’s vSAN license. If enabling DiT fails with “Host(s) don’t have data-in-transit encryption license”, follow KB 373324.

5. Align the DiT setting

Option A: from vCenter (recommended when vCenter is available)

Go to cluster › Configure › vSAN › Services › Data-In-Transit Encryption › EDIT.
Set the intended state and rekey interval, then APPLY. vCenter pushes the setting to every connected host.
If the cluster already shows the intended state, turn DiT off, APPLY, then turn it on again and APPLY, so vCenter re-pushes the setting.

Option B: from the host (when vCenter is unavailable, or for one host)

On each mismatched host, set DiT to match the cluster. To enable:

esxcli vsan network security set -e true

To disable:

esxcli vsan network security set -e false

Run esxcli vsan network security set --help to see the option for the rekey interval, and set it to match the cluster.

Check the change:

esxcli vsan network security get

If the Configuration check still reports a mismatch after the hosts agree, restart the management agents on each host, one host at a time:

/etc/init.d/vsanmgmtd restart

/etc/init.d/hostd restart

6. Validate

On every host, esxcli vsan cluster get shows Sub-Cluster Member Count equal to the number of hosts in the cluster.
On every host, esxcli vsan network security get reports the same status.
Check object health:

esxcli vsan debug object health summary get

There should be no inaccessible objects. A resync may run for a while after the partition heals.

vSAN Health › RETEST: vSAN cluster partition and Data-in-transit encryption - Configuration check are green.

Additional Information

Related articles

Troubleshooting vSAN Encryption - KMS KB 326769
Troubleshooting vSAN Encryption - NKP KB 448917
vSAN Health Service - Data-in-transit encryption Health – Configuration check KB 315521
vSAN health alert “Configuration Check” triggered and cluster partition noticed KB 418748
Data-in-transit encryption is disabled on this cluster but enabled on this host KB 373324
vSAN node may experience cluster partition and/or data unable to resync due to corruption in transit of CMMDS and/or RDT data packets KB 414383
How to enable/disable encryption on vSAN cluster KB 400462

Documentation

vSAN Data-In-Transit Encryption VCF 9.0 documentation
vSAN Encryption Services vmware.com