Virtual Machine with attached Encrypted CNS Volumes failed to restart via vSphere HA Failover

Products

• VMware vCenter Server

Issue/Introduction

In a vSphere environment using encrypted Cloud Native Storage (CNS) / First Class Disk (FCD) volumes, vSphere HA may fail to restart a Kubernetes or Supervisor-managed virtual machine after the ESXi host running the VM becomes unavailable.

The following symptoms be observed:

- An ESXi host fails or becomes unavailable.
- vSphere HA initiates failover for the affected VM.
- HA cannot find a compatible surviving ESXi host.
- The VM remains powered off, resulting in extended application downtime.
- The encrypted CNS/FCD volume encryption key is present only on the ESXi host where the disk was originally attached.
- Other hosts in the HA/DRS cluster do not have the required encryption key.

• The FDM log may report:
  
  
  • Host host-###### changed state: Dead
  • Dump Reason=FailoverStart
  • Compat list for .../<VM>.vmx: host-######
    
    
• The compatibility list contain only the failed ESXi host.
  
  Subsequent placement attempts can fail with:
  
  
  • Vm ... failed placement with fault
  • vim.fault.NoActiveHostInCluster
  • Setting insufficient resource timeout to 60 seconds and retrying placement
    
    
• The vCenter vpxd.log report that the encryption key is missing from the surviving ESXi hosts:
  
  
  • [sub=CryptoManager]
  • Key <Key-ID>/<Key-Provider> is not present at host
  • [vim.HostSystem:host-#####,host2.example.com].
    
    
  • [sub=CryptoManager]
  • Key <Key-ID>/<Key-Provider> is not present at host
  • [vim.HostSystem:host-######,host3.example.com]
    
    [sub=CryptoManager]
  • Key <Key-ID>/<Key-Provider> is not present at host
  • [vim.HostSystem:host-######,host4.example.com]
    
    
• DRS subsequently return no placement recommendation:
  
  
  • [sub=DAS]
  • Invoking DRS for VM Placement recommendation;
  • type: VmFailover
    
    
  • [sub=DAS]
  • Last Pass - No DRS Recommendation generated.

Environment

- vCenter Server 8.0 3h
- vSphere ESXi 8.0 3i
- Supervisor version 1.30

Cause

This issue is caused by a code defect in the encrypted CNS/FCD disk attach workflow.

When an encrypted CNS/FCD disk is attached to a VM, the affected workflow sends the disk encryption key only to the ESXi host currently running the VM.

The key is registered as a host-level key rather than as a cluster-level key.

As a result, the encryption key is not automatically distributed to the other ESXi hosts participating in the HA/DRS cluster.

Resolution

The VMware engineering team is aware of this issue and is working to on a fix. 

 

 Workaround

To remediate an affected VM and propagate its encryption key to the other hosts in the cluster, perform a manual vMotion of the VM to an ESXi host that does not already have the encryption key.

The vMotion workflow uses a different encryption-key handling path. During migration, the disk encryption key is promoted to a cluster-level key and distributed to the applicable hosts in the HA/DRS cluster.

Procedure:

1. Identify the VM using the affected encrypted CNS/FCD volume.

2. Confirm that other ESXi hosts are reporting the encryption key as missing in vpxd.log.

Example:

Key <Key-ID>/<Key-Provider> is not present at host <ESXi-host>

3. Ensure the affected VM is running on a host that currently has access to the encryption key.

4. Perform a manual compute vMotion of the VM to another healthy ESXi host in the same HA/DRS cluster that previously reported the key as missing.

5. Allow the vMotion operation to complete successfully.

6. Verify vpxd.log and confirm that the previous missing-key warnings for the affected encryption key no longer occur for the cluster hosts.