Alert: Host requires encryption mode enabled. Manually recover the missing key

Products

VMware vCenter Server VMware Cloud Foundation 4.x VMware vCenter Server 6.0 VMware vCenter Server 7.0 VMware vCenter Server 8.0 VMware vSphere ESXi VMware vSphere ESX 6.x VMware vSphere ESX 7.x VMware vSphere ESX 8.x

Issue/Introduction

Either after a reboot of an ESXi host with Host Encryption enabled, after attempting to enable Host Encryption Mode, or following an ESXi host upgrade, the ESXi host reports an alert

Host requires encryption mode enabled. Manually recover the missing key

Host <host FQDN> requires encryption mode enabled. Check the status of the key provider <host name> and manually recover the missing key <host key name> to the key provider <KMS>. Go to https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-security-7-0/use-encryption-in-your-vsphere-environment/re-enable-esxi-host-encryption-mode.html for detailed remediation steps.

There are no connection issues between the vCenter and the KMS, and all other hosts are still using encryption normally.

Environment

6.x, 7.x, 8.x

Cause

This is due to a mismatch in the information between the ESXi Host and the vCenter.
The vCenter information points to the ESXi host still having the Host Key available, but the host does not have access to the key following the reboot.
Due to the this stale information, the ESXi host is unable to get a new Host Key from the KMS through the vCenter.

Resolution

Take offline snapshots of all vCenter Servers in ELM. Refer VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice
Place affected ESXi host into Maintenance Mode.
SSH to the ESXi host with root user

Run the following command to ensure the Host does not have a Host Key

crypto-util keys getkidbyname "HostKey"

Sample output

crypto-util keys: A key for 'HostKey' has not been established

SSH to vCenter with root user.
Stop the vpxd service
```
service-control --stop vmware-vpxd
```
Connect to the VCDB via the postgres shell. Refer Interacting with the vCenter Server Appliance 6.5/6.7/7.0/8.0 embedded vPostgres Database
```
psql -U postgres -d VCDB -h localhost
```

Identify the information for the host in question

select id, dns_name, crypto_state, crypto_key_id, crypto_key_provider_id, crypto_enable from vpx_host where dns_name like '%<esxi-host-fqdn>%';

Sample output

VCDB=# select id, dns_name, crypto_state, crypto_key_id, crypto_key_provider_id, crypto_enable from vpx_host where dns_name like '%esxi01.example.com%';

-[ RECORD 1 ]----------+---------------------------------------------------------------------------------------------------------------------------------------------
id                     | 0000
dns_name               | esxi01.example.com
crypto_state           | 0
crypto_key_id          |  ASE#######EA
crypto_key_provider_id | ######
crypto_enable          | 1

Using the id for the affected host, run the following command to clear its stale information

update vpx_host set crypto_state=0, crypto_enable=0, crypto_key_id='', crypto_key_provider_id='' where id=<host id from step# 8>;

Sample output after updating DB

VCDB=# select id, dns_name, crypto_state, crypto_key_id, crypto_key_provider_id, crypto_enable from vpx_host where dns_name like '%esxi01.example.com%';

id                     | 0000
dns_name               | esxi01.example.com
crypto_state           | 2
crypto_key_id          | ASE#######EA
crypto_key_provider_id | ######
crypto_enable          | 1

Quit the postgres shell
```
\q
```
start vpxd service
```
service-control --start vmware-vpxd
```
Login to the vSphere UI and confirm that the affected host now no longer shows the alarm, and the host encryption is enabled (If not already enabled, we can now enable it manually.)

Additional Information

The steps involving making manual edits to the vCenter DB. Any incorrect steps or changes to the DB can result in a catastrophic failure of the vCenter.
This is further exacerbated by the fact that we are making edits related to encryption keys - any incorrect steps or changes can lead to unrecoverable VMs.
Please take offline snapshots of all vCenter Servers in Enhanced Linked Mode before proceeding.