Troubleshooting vSAN Encryption - NKP
search cancel

Troubleshooting vSAN Encryption - NKP

book

Article ID: 448917

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Provides guidance for resolving vSAN encryption failures using Native Key Provider (NKP). Symptoms include the inability to configure encryption, encrypted disks failing to mount, and vSAN health warnings regarding vSAN Encryption configuration.

Please note: Ensure vCenter resides outside the vSAN datastore if encryption is enabled; running vCenter on the same encrypted cluster is unsupported resulting in a circular redundancy. 

[!WARNING] If the vSAN cluster is encrypted and not using TPM, you must NOT reboot any configured ESXi hosts if the Key Provider (NKP) is missing or unhealthy.

Encryption keys remain in memory until a reboot, so refraining from rebooting is critical to maintaining data access. See vSAN Encryption Services page 12.

NKP doesn't require to establish trust to a KMIP server like a KMS does as it resides within vCenter. So it's expected behavior to see the key/cert files under "/etc/vmware/ssl/" to be 0 bytes in size.

Example:

-rw-r--r-T    1 root     root           0 Jan  1  2000 vsan_kms_castore.pem
-rw-r--r-T    1 root     root           0 Jan  1  2000 vsan_kms_castore_old.pem
-rw-r--r-T    1 root     root           0 Jan  1  2000 vsan_kms_client.crt
-r-------T    1 root     root           0 Jan  1  2000 vsan_kms_client.key
-rw-r--r-T    1 root     root           0 Jan  1  2000 vsan_kms_client_old.crt
-r-------T    1 root     root           0 Jan  1  2000 vsan_kms_client_old.key
-rw-r--r-T    1 root     root           0 Jan  1  2000 vsanvp_castore.pem

Ensure to have a backup of the NKP as it ensures faster data recovery if there is an issue with the NKP.

Environment

VMware vSAN

Encryption enabled using NKP

Cause

Encryption issues with an NKP typically stem from the NKP in an unhealthy state or accidently deleted from vCenter preventing the host's inability to retrieve its unique HostKey.

Resolution

Troubleshooting vSAN Encryption

Checklist

  • For initial configuration of vSAN Encryption, the vCenter and the ESXI hosts in the cluster will require connectivity, but for ongoing operation, only the hosts require it. vCenter is only required when configuration needs to be changed or when enabling/disabling Encryption.
  • Ensure the host can enter crypto-safe mode. To do this, it requires access to its HostKey, which is a different key to that required to mount any encrypted diskgroups. Without this key, the host is not deemed secure enough to host encrypted disk groups or VMs.
  • Ensure the host has access to and can retrieve the required vSAN KDK.
  • If the NKP is in an unhealthy state or missing from vCenter restore it from backup to restore proper functionality

 

Commands to verify the configured NKP servers and status on ESXI hosts.

  • Get vSAN encryption information.

esxcli vsan encryption info get                

  • List the KMS configurations used for vSAN encryption.

esxcli vsan encryption kms list                                 

  • Get host key from keycache used for vSAN encryption.

esxcli vsan encryption hostkey get

 

Review NKP configuration on ESXI hosts.

To get the NKP info from a host using a Native Key Provider run the below commands:

[root@hostname:~] configstorecli config current get -c esx -g trusted_infrastructure -k "kms_providers"
[
{
     "native_provider": {
        "key_derivation_key": "*******",
        "key_id": "########-####-####-####-############"
     },
     "provider": "<name>",
     "type": "NATIVE"
  }
]
  "providers": [
     {
        "key_server": {
           "connection_timeout": -1,
           "kmip_server": {
              "servers": [
                 {
                    "hostname": "<providername>:kmx",
                    "name": "NativeKeyProvider",
                    "port": 0
                 }
              ],
              "username": ""
           },
           "proxy_server": {
              "hostname": "",
              "port": -1
           },
           "type": "KMIP"
        },
        "master_key_id": "<keyid>",
        "old": false,
        "provider": "NativeKeyProvider"
      }


Note: Output modified to show only the relevant encryption information.

  • (The below output has been modified to only show the related encryption info. The full output is around several hundred lines hence why it's piped to less.)

[root@hostname:~] configstorecli config current get -c vsan -g system -k "vsan"|less
{
   "data_efficiency": {
      "compression_enabled": true,
      "dedup_enabled": false
   },
   "default_datastore_name": "vsanDatastore",
   "enabled": true,
   "encryption": {
      "changing": false,
      "dek_generation_id": 1,
      "dek_id": "#############",
      "dek_verifier": "+Ylf9O0cmkI=",
      "enabled": true,
      "erase_disks_before_use": false,
      "host_key_id": "#############",
      "initialization_vector": "#############",
      "kek_id": "#############",
      "kek_verifier": "#############",
      "kmip_cluster_id": "TestNKP",
      "old_dek_id": "",
      "old_dek_verifier": "",
      "old_wrapped_dek": "",
      "syncing": false,
      "wrapped_dek": "#############"
   },
   "host": {
      "generation_num": 6,
      "generation_time": 1785859641563,
      "node": "DATA"
   },
   "in_transit_encryption": {
      "enabled": true,
      "rekey_interval": ###,
      "state": "SETTLED"
   },
   },
   "providers": [
      {
         "key_server": {
            "connection_timeout": -1,
            "kmip_server": {
               "servers": [
                  {
                     "hostname": "nkp:kmx",
                     "name": "NativeKeyProvider",
                     "port": 0
                  }
               ],
               "username": ""
            },
            "proxy_server": {
               "hostname": "",
               "port": -1
            },
            "type": "KMIP"
         },
         "master_key_id": "TestNKP",
         "old": false,
         "provider": "NativeKeyProvider"
      }

Note: The preceding command output excerpts are only examples. Environmental variables may vary depending on your environment.

Ensure the host can enter crypto-safe mode.

Verify the host is in crypto-safe mode.

  • To enter crypto-safe mode, the host must be able to retrieve a special key called the HostKey.

  • This key is separate from any other keys that would be required to encrypt VMs or the vSAN datastore.  This key is used by the host to encrypt core dumps. Without access to this key, the host will be unable to request any other keys from the NKP, even if it is accessible.

  • When vSAN Encryption was first enabled on the cluster, the host transitioned to ‘crypto-safe’ mode for the first time and was assigned a key to install as its HostKey. The host will always look for this key, based on the key identifier, when booting up. The host will NOT attempt to retrieve, nor will it request, a different key if the original key is not available. So for the host to re-enter crypto-safe, this key MUST be available.

  • To determine if a HostKey has been installed (i.e. the host is crypto-safe), you can use the UI (if available).

  • Check that Encryption Mode is enabled.  If it is not, attempt to enable through the UI. 

  • If the host will not enter encryption mode, then it cannot retrieve its HostKey.

  •  If the UI is not available, you can use the crypto-util utility on the host to see if a HostKey has been installed or not.

    [root@hostname:~] crypto-util keys getkidbyname HostKey
    vmware:key/fqid/<VMWARE-NULL>/HyTrust/04f631cc%2d84dd%2d11e8%2d8194%2d00505698ddb6

  • If a key value is returned, the host is in crypto-safe mode. If the message indicates that a HostKey has not been established, then the host is not in crypto-safe mode.

  • To investigate which key the host requires to enter crypto-safe mode, you can find this value by looking in the vCenter MOB. (The host MOB is no longer available but can be accessed via vCenter).

    Navigate to the host page in the MOB: https://vcsa.domain.local/mob/?moid=host-##### for example.

To find the host-id, please navigate to vCenter server and click on the hostname on Cluster. You can find the host-id as highlighted below.

  • Navigate to the host page in the MOB: https://vcsa.domain.local/mob/?moid=host-###### for example.
    1. Click Runtime

     2. Click CryptoKeyId

As seen above the UUID of the key the host will require to enter crypto-safe mode.

    3.  Click ProviderId.

Log Identification:

If vCenter is not available the HostKey identifier can only be gathered through log review.

In Hostd.log:

  • Grep for the term ‘CryptoManager’ in the /var/log/hostd.log to see the host adding keys to its keyCache. For example, a host logging the following when it successfully added the HostKey to the cache:

2026-09-19T18:04:39.643Z -INFO Hostd 1000342665 [esx@4413 sub="Hostsvc.CryptoManager" opID="b02a0629" sid="52e60d87" user="dcui:vsanmgmtd"] Host has been placed in Crypto-prepared state
2026-09-19T18:04:39.645Z -INFO Hostd 1000342665 [esx@4413 sub="Dispatcher" opID="b02a0629" sid="52e60d87" user="dcui:vsanmgmtd"] Running [1] : vim.encryption.CryptoManagerHost.prepare,
2026-09-19T18:04:39.646Z -INFO Hostd 1000342665 [esx@4413 sub="Vimsvc.TaskManager" opID="b02a0629" sid="52e60d87" user="dcui:vsanmgmtd"] Task Completed : haTask-ha-host-vim.encryption.CryptoManagerHost.prepare-2021170876 Status success
2026-09-19T18:04:39.889Z -DEBUG Hostd 1000342649 [esx@4413 sub="Hostsvc.CryptoManager" opID="b02a062f"] Retrieved host key #############//#############/TestNKP from key cache/TestNKP from key cache
2026-09-19T18:18:30.860Z -DEBUG Hostd 1000342639 [esx@4413 sub="Hostsvc.CryptoManager" opID="00babcac-ff2f-0821" sid="524ca04b" user="dcui:hcid"] Retrieved host key #############/#############//#############/TestNKP from key cache

  • Example when a host can't retrieve a HostKey:

    2026-09-19T21:02:34.887Z -DEBUG Hostd 1000343577 [esx@4413 sub="Hostsvc.CryptoManager" opID="20f43ed6" sid="52e2c2fd" user="dcui:vsanmgmtd"] No hostKey set in key cahce.
    2026-09-19T21:02:43.336Z -DEBUG Hostd 1000343570 [esx@4413 sub="Hostsvc.CryptoManager" opID="20f43ed7"] No hostKey set in key cahce.

In vmkernel.log:

  • If the host was rebooted and a TPM is not in use then the disks will not mount and the below behavior is seen:

    esxcli vsan storagepool list
    mpx.vmhba0:##:##:##
       Device: mpx.vmhba0:##:##:##
       Display Name: mpx.vmhba0:##:##:##
       vSAN UUID: 5228d8fd-e677-1f23-16a6-############
       Used by this host: false
       In CMMDS: false
       On-disk format version: 21
       Checksum: ##############
       Checksum Ok: true
       Is Mounted: false
       Is Encrypted: true
       Disk Type: singleTier
       Creation Time: Sat Sep 19 18:47:05 2026

    mpx.vmhba0:##:##:##
       Device: mpx.vmhba0:##:##:##
       Display Name: mpx.vmhba0:##:##:##
       vSAN UUID: 52578651-4d19-3b89-3955-############
       Used by this host: false
       In CMMDS: false
       On-disk format version: 21
       Checksum: ##############
       Checksum Ok: true
       Is Mounted: false
       Is Encrypted: true
       Disk Type: singleTier
       Creation Time: Sat Sep 19 18:48:56 2026


    2026-09-19T20:47:01.926Z -WARNING vmkwarning - [user1@####-HOST01] cpu12:1000341729) PLOG: PLOGValidateDisk:3611: Disk mpx.vmhbaX:CX:TX:L1:1 0x468fc0020cc8 is unhealthy state (2), degraded


Additional Information

vSphere Native Key Provider (NKP) Questions & Answers

Troubleshooting vSAN Encryption - KMS

 

Powered by Wolken Software