Powered off Windows VMs randomly locked after the reboot of hosts with Native key provider.

Products

• VMware Cloud Foundation
• VMware vCenter Server

Issue/Introduction

Symptoms:

Shutdown Windows VMs randomly gets locked after the host power cycle. 



(1) Entering maintenance mode:

2024-02-21T13:54:16.325+09:30 info vpxd[07867] [Originator@6876 sub=MoCluster] HDCS mgr Excluding host for placing HDCS VM [vim.HostSystem:host-1048,vcenter_FQDN Reason: Maintenance mode ENABLED

 

(2) Host was powered off:

2024-02-21T13:55:34.861+09:30 info vpxd[07918] [Originator@6876 sub=MoHost opID=HB-host-1048@457-1ee3fc00] Host [vim.HostSystem:host-1048,vcenter_FQDN hardware ID string: |AssetTag:||ServiceTag:XXXX||EnclosureSerialNumberTag:XXX||SerialNumberTag:XXX||OemSpecificString:Dell System||OemSpecificString:5[0000]||OemSpecificString:14[1]||OemSpecificString:17[06A5CEFB3CF99F58]||OemSpecificString:17[06A309FCA09FD603]||OemSpecificString:18[0]||OemSpecificString:19[1]||OemSpecificString:19[1]||OemSpecificString:26[0]||OemSpecificString:31[1]||OemSpecificString:30[0000000069BD4000;00100000]|

2024-02-21T13:55:34.881+09:30 info vpxd[07918] [Originator@6876 sub=QuickStats opID=HB-host-1048@457-1ee3fc00] Host [vim.HostSystem:host-1048 vcenter_FQDN should not be polled

2024-02-21T13:55:34.889+09:30 info vpxd[07918] [Originator@6876 sub=MoCluster opID=HB-host-1048@457-1ee3fc00] HDCS mgr Excluding host for placing HDCS VM [vim.HostSystem:host-1048 vcenter_FQDN Reason: Maintenance mode ENABLED

 

(3) VM -VM Name was locked:

2024-02-21T13:55:34.942+09:30 info vpxd[07918] [Originator@6876 sub=CryptoManager opID=HB-host-1048@457-1ee3fc00] The crypto state of VM [vim.VirtualMachine:vm-49313,VM_NAME] on host [vim.HostSystem:host-1048,vcenter_FQDN was changed from unlocked to locked

 

(4) Key fetching failed:

2024-02-21T13:56:34.902+09:30 warning vpxd[06834] [Originator@6876 sub=CryptoManager opID=HB-host-1048@457-1ee3fc00] Failed to get key XXX - Primary: key provider D71 - Primary not managed by VC.

 

The error that the native key provider 'D71 - Primary' was not managed by VC is weird because the manual unlock succeeded:

 

2024-02-21T14:11:00.279+09:30 info vpxd[07845] [Originator@6876 sub=CryptoManager opID=lntsrkrw-660365-auto-e5ji-h5:70055337-19] Sending keys to unlock VM [vim.VirtualMachine:vm-49313,VM_NAME], user AD/username, VM host: [vim.HostSystem:host-1048,VC_FQDN], cluster hosts: [[vim.HostSystem:host-1048,VC_FQDN], [vim.HostSystem:host-1045,vcenter_FQDN [vim.HostSystem:host-1050,vcenter_FQDN

2024-02-21T14:11:00.552+09:30 info vpxd[06749] [Originator@6876 sub=MoCluster opID=lntsrkrw-660365-auto-e5ji-h5:70055337-19-36de54c0] Conservatively reserve one HDCS slot on host [vim.HostSystem:host-1048 vcenter_FQDN

2024-02-21T14:11:00.558+09:30 info vpxd[06749] [Originator@6876 sub=CryptoManager opID=lntsrkrw-660365-auto-e5ji-h5:70055337-19-36de54c0] The crypto state of VM [vim.VirtualMachine:vm-49313,VM_NAME] on host [vim.HostSystem:host-1048,vcenter_FQDN was changed from locked to unlocked 

 

Environment

VMware vSphere 7.0.x
VMware vCenter Server 7.0.0
VMware Cloud Foundation 4.x
VMware vCenter Server 7.0.3
VMware vSphere ESXi 7.0
VMware vCenter Server 7.0.2
VMware vCenter Server 7.0.1
VMware vCenter Server 7.0.x

Cause

After the reboot of the ESXI Host , Powered off Windows VM will go to lock state it is due to the host configured with Native key provider.

 

Resolution

No resolution yet. The fix expected in the future release of vCenter. 

Workaround:

We can manually unlock the VM. If there is multiple VM then please follow the below power cli command.

      

         1. Using below powerCLI cmdlet to unlock all the VMs in the VC:

    Get-VM|Where-Object {$_.ExtensionData.Runtime.CryptoState -eq 'locked'} | Unlock-VM

    This cmdlet can get all locked VM and unlock them together.

 

 

Additional Information

Impact/Risks:

Shutdown VMs will get lock state and need to unlock manually.