VM showing as invalid and asking to be "unlocked"

Products

• VMware vCenter Server

Issue/Introduction

• Virtual machines with a vTPM configured are displayed as "invalid" in vSphere Client
• In addition a "Virtual machine locked" alarm is triggered
• Attempts to unlock the VM fail and errors with a message “A general system error occurred: Unable to decrypted the ciphertext. Failed to decrypted the key.....”
• the following error stack can be found in the vpxd logs of the vCenter Server under /var/log/vmware/vpxd:

  <timestamp> info vpxd[32071] [Originator@6876 sub=vmomi.soapStub[465672] opID=HB-host-####@<task-id>] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007fb108047cd8, h:20, <TCP '#.#.#.# : 41686'>, <TCP '#.#.#.# : 443'>>), /sdk>, method: configureCryptoKey; code: 500(Internal Server Error); fault: (vmodl.fault.SystemError) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = (vmodl.LocalizableMessage) [
  -->       (vmodl.LocalizableMessage) {
  -->          key = "com.vmware.esx.kms.key_operation.decrypt.error",
  -->          arg = (vmodl.KeyAnyValue) [
  -->             (vmodl.KeyAnyValue) {
  -->                key = "arg",
  -->                value = "###########################################################################################"
  -->             }
  -->          ],
  -->          message = "Failed to decrypt the key '###########################################################################################'."
  -->       },
  -->       (vmodl.LocalizableMessage) {
  -->          key = "com.vmware.esx.kms.keyoperationsvc.decrypt.error",
  -->          arg = <unset>,
  -->          message = "Unable to decrypt the ciphertext."
  -->       }
  -->    ],
  -->    reason = ""
  -->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007fb108047cd8, h:20, <TCP '#.#.#.# : 41686'>, <TCP '#.#.#.# : 443'>>), /sdk>]: configureCryptoKey
  --> A general system error occurred: "
  --> }

• the hostd logs on the ESXi hosting the virtual machine, found in /var/run/log, contain the following errors:

  <timestamp> warning hostd[2100375] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-###-###########/<vm-folder>/<vm-name>
  .vmx opID=m2q87oru-2789342-auto-1ns9r-h5:70063624-33-01-33-81c4 user=vpxuser:######\#######] Get trusted platform keys failed: N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError
  --> )
  <timestamp> error hostd[2101633] [Originator@6876 sub=Hostsvc.CryptoManager opID=######## user=vpxuser] Failed to invoke "com.vmware.esx.kms.key_operation.decrypt" : Error:
  -->    com.vmware.vapi.std.errors.error
  --> Messages:
  -->    com.vmware.esx.kms.keyoperationsvc.decrypt.error<Unable to decrypt the ciphertext.>
  <timestamp> verbose hostd[2101633] [Originator@6876 sub=Solo.Vmomi opID=######## user=vpxuser] Arg keyId:
  --> (vim.encryption.CryptoKeyId) {
  -->    keyId = "#########################################################################",
  -->    providerId = (vim.encryption.KeyProviderId) {
  -->       id = "<vm-id>"
  -->    }
  --> }
  <timestamp> info hostd[2101633] [Originator@6876 sub=Solo.Vmomi opID=######## user=vpxuser] Throw vmodl.fault.SystemError
  <timestamp> info hostd[2101633] [Originator@6876 sub=Solo.Vmomi opID=######## user=vpxuser] Result:
  --> (vmodl.fault.SystemError) {
  -->    faultMessage = (vmodl.LocalizableMessage) [
  -->       (vmodl.LocalizableMessage) {
  -->          key = "com.vmware.esx.kms.key_operation.decrypt.error",
  -->          arg = (vmodl.KeyAnyValue) [
  -->             (vmodl.KeyAnyValue) {
  -->                key = "arg",
  -->                value = "############################################################/######"
  -->             }
  -->          ],
  -->       },
  -->       (vmodl.LocalizableMessage) {
  -->          key = "com.vmware.esx.kms.keyoperationsvc.decrypt.error",
  -->       }
  -->    ],
  -->    reason = "",
  -->    msg = ""
  --> }

Environment

VMware vSphere ESXi 7.0.x

VMware vSphere ESXi 8.0.x

Cause

• The virtual machines .vmx configuration file is encrypted, but key was provided by a different key provider than the one currently registered with vCenter Server
• This can happen when the native key provider in vCenter was replaced or changed

Resolution

To fix this issue:

1. 1. Unregister the VM from inventory.
      
      
   2. Create a new vmx file and match the vmx configuration settings to the source VM, but do not add any entries for vTPM.
      
      Alternatively, you can take a backup of existing vmx file and edit the entries for vtpm as below:
      
      vtpm.ekCSR = ""
vtpm.ekCRT = ""
migrate.encryptionMode = ""
ftcpt.ftEncryptionMode = ""
encryption.keySafe = ""
encryption.data = ""


   3. Register the VM.
      
      
   4. Edit the VM settings to add the vTPM.
      

      vtpm.present = "True"

   5. Power on the VM.

Additional Information

Deactivate Host Encryption Mode Using the API

Unlock Locked Virtual Machines

Using Encryption in Your vSphere Environment

Resolve Missing Encryption Key Issues