.vmx file of an encrypted VM is decrypted after a scheduled VM compatibility upgrade

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• After a scheduled VM compatibility upgrade of an encrypted VM, the .vmx file will have the non-empty line for 'dataFileKey’ field as shown here:

$ cat <VM-Name>.vmx | grep 'dataFileKey' | wc -l
1

$ cat <VM-Name>.vmx | grep 'virtualHW.scheduledUpgrade.state'
virtualHW.scheduledUpgrade.state = "done"

Note: Only the encrypted VM Home (.vmx file) is affected. Any encrypted disks associated with that VM remain unaffected (encrypted). This issue does not impact unscheduled VM Compatibility upgrades of encrypted VMs.

Any subsequent reconfiguration tasks on this VM may result in errors. For example, adding a CD/DVD drive to this VM will result in
"Virtual machine has encryption enabled, but the virtual machine home doesn't have an encryption profile or doesn't have a configured security device (like TPM)".

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.0

Cause

A bug in scheduled VM compatibility upgrade of encrypted VM results in .vmx file being written in unencrypted state.

Resolution

This issue is resolved in vSphere 6.5 U3 / vSphere 6.7 U2 / vSphere 7.0. available at VMware download .

Workaround:
To workaround this issue:

There are two ways to manually remediate the affected VM. But, it is advised to upgrade to recommended patch release.

Solution 1: [Re-apply storage policy]

1. Ensure the VM is powered-off.

 Note: Make a note of the current configuration of each disk associated with the VM (i.e. whether encrypted or not).

2. Unregister the VM from vCenter. 
3. Re-register the VM in vCenter back to the same host as before.
4. Re-apply the  encryption policy (e.g. “VM Encryption Policy”) to the VM Home and to each of the disk which are intended to be encrypted.

 

Note:  The .vmx file now must be re-encrypted. Specifically, 'dataFileKey' should not be present and instead we must have 'encrypted.keySafe'. The .vmx file uses a new/different key that the one that was used before.

Solution 2: [Move all disks]

1. Ensure the VM is powered-off.
2. Create a new encrypted VM with no disks.

Note:similar to the original VM.

3. Remove all the disks from the old VM, and attach them to the new VM. The storage policy of each disk must remain the same as before.

Note:  The .vmx file now must be re-encrypted. Specifically, 'dataFileKey' should not be present and instead we must have 'encrypted.keySafe'. The .vmx file uses a new/different key that the one that was used before.