vCenter Error: General vSAN error. There was an issue generating keys with KMS cluster

Products

• VMware vCenter Server

Issue/Introduction

When attempting to enable Data-at-Rest Encryption on a vSAN cluster, the process fails with the following error in vCenter:

General vSAN error. There was an issue generating keys with KMS cluster.

Environment

VMware vCenter Server: 8.0 / 7.0.x
VMware ESXi: 8.0 / 7.0.x
vSAN: 8.0 / 7.0.x
External Key Management Server (KMS): Any KMIP-compliant provider

Cause

The issue is typically caused by network or firewall restrictions blocking TCP Port 5696. This port is required for the Key Management Interoperability Protocol (KMIP), which ESXi hosts use to communicate with the KMS cluster to generate and retrieve encryption keys.

Resolution

To resolve this issue, ensure that bidirectional communication is allowed on port 5696 between all ESXi hosts in the cluster and the KMS server IPs.

1. Verify Port Connectivity
Log in to the ESXi host via SSH and run the following command to test connectivity to the KMS server:

nc -zv <KMS_IP_Address> 5696

If the output does not indicate a successful connection (e.g., Connection refused or Timeout), the port is blocked.

2. Coordinate with Network Team
The customer should engage their internal network or security team to:

    1. Confirm that firewall rules permit TCP traffic on Port 5696.
    2. Ensure there are no intermediate devices (IPS/IDS) dropping KMIP traffic.
    3. Verify that the ESXi management or vSAN VMkernel interfaces have a valid route to the KMS network.

3. Once the port is confirmed open, re-attempt to enable encryption from the vCenter UI.

Additional Information

Note: While the Key Management Server may be reachable via ICMP (ping), the specific KMIP traffic on port 5696 can still be dropped by the customer's internal network security policies.

Perform Packet Capture
If the network team reports the port is open but connectivity still fails, use the pktcap-uw tool to verify if packets are leaving the VMkernel interface:

pktcap-uw --vmknic vmk0 --port 5696