Virtual machine displays an "Invalid" state in vCenter Server and cannot be unlocked

Products

• VMware vCenter Server

Issue/Introduction

• Virtual machines with a vTPM configured are displayed as "invalid" in vSphere Client.
  
  
• In addition, a "Virtual machine locked" alarm is triggered.
  
  
• Attempts to unlock the virtual machine fail and errors with a message: A general system error occurred: Unable to decrypt the ciphertext. Failed to decrypt the key '##########################'.
  
  
• The following error stack can be found in /var/log/vmware/vpxd/vpxd.log of the vCenter Server:
  <timestamp> info vpxd[32071] [Originator@6876 sub=vmomi.soapStub[465672] opID=HB-host-####@<task-id>] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007fb108047cd8, h:20, <TCP '#.#.#.# : 41686'>, <TCP '#.#.#.# : 443'>>), /sdk>, method: configureCryptoKey; code: 500(Internal Server Error); fault: (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.esx.kms.key_operation.decrypt.error",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "arg",
-->                value = "###########################################################################################"
-->             }
-->          ],
-->          message = "Failed to decrypt the key '###########################################################################################'."
-->       },
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.esx.kms.keyoperationsvc.decrypt.error",
-->          arg = <unset>,
-->          message = "Unable to decrypt the ciphertext."
-->       }
-->    ],
-->    reason = ""
-->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007fb108047cd8, h:20, <TCP '#.#.#.# : 41686'>, <TCP '#.#.#.# : 443'>>), /sdk>]: configureCryptoKey
--> A general system error occurred: "
--> }
  
  
• The /var/run/log/hostd.log on the ESXi host running the virtual machine contains the following errors:
  <timestamp> warning hostd[2100375] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-###-###########/<vm-folder>/<vm-name>.vmx opID=########-#######-auto-#####-h5:######-##-##-##-#### user=vpxuser:######\#######] Get trusted platform keys failed: N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError--> )
<timestamp> error hostd[2101633] [Originator@6876 sub=Hostsvc.CryptoManager opID=######## user=vpxuser] Failed to invoke "com.vmware.esx.kms.key_operation.decrypt" : Error:
-->    com.vmware.vapi.std.errors.error
--> Messages:
-->    com.vmware.esx.kms.keyoperationsvc.decrypt.error<Unable to decrypt the ciphertext.>
<timestamp> verbose hostd[2101633] [Originator@6876 sub=Solo.Vmomi opID=######## user=vpxuser] Arg keyId:
--> (vim.encryption.CryptoKeyId) {
-->    keyId = "#########################################################################",
-->    providerId = (vim.encryption.KeyProviderId) {
-->       id = "<vm-id>"
-->    }
--> }
<timestamp> info hostd[2101633] [Originator@6876 sub=Solo.Vmomi opID=######## user=vpxuser] Throw vmodl.fault.SystemError
<timestamp> info hostd[2101633] [Originator@6876 sub=Solo.Vmomi opID=######## user=vpxuser] Result:
--> (vmodl.fault.SystemError) {
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.esx.kms.key_operation.decrypt.error",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "arg",
-->                value = "############################################################/######"
-->             }
-->          ],
-->       },
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.esx.kms.keyoperationsvc.decrypt.error",
-->       }
-->    ],
-->    reason = "",
-->    msg = ""
--> }

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 7.x
vCenter Server 8.x
vCenter Server 7.x

Cause

• This issue occurs because the virtual machine configuration (.vmx) file is encrypted with a key from a provider that is no longer registered with the vCenter Server.
• This typically happens after the native key provider has been replaced or modified.

Resolution

1. Unregister the virtual machine (VM) from the vCenter inventory.
   
   
2. Modify the virtual machine configuration file (.vmx) using one of the following methods:
   Option A: Create a new .vmx file matching the source VM settings, excluding all vTPM entries.
   Option B: Back up the existing .vmx file and clear the following entries by setting them to empty quotes "".
   vtpm.ekCSR = ""
vtpm.ekCRT = ""
migrate.encryptionMode = ""
ftcpt.ftEncryptionMode = ""
encryption.keySafe = ""
encryption.data = ""
   
   
3. Register the virtual machine back into the vCenter inventory.
   
   
4. Edit the VM Settings and add a new vTPM device (this sets vtpm.present = "True").
   
   
5. Power on the virtual machine.

Additional Information

Deactivate Host Encryption Mode Using the API

Unlock Locked Virtual Machines

Using Encryption in Your vSphere Environment

Resolve Missing Encryption Key Issues