vSAN encryption DEK generation ID mismatch after upgrade to vSAN7.0.

Products

• VMware vSAN

Issue/Introduction

Symptoms:

vSAN health reports a vSAN cluster configuration consistency issue "Data is encrypted with an out of date Data Encryption Key", as seen in the below screenshot.

Environment

VMware vSAN 7.0.x

Cause

Prior to 7.0 GA the config file esx.conf was used to persist ESXi configurations including vSAN encryption info. Once the vSAN encryption is turned on, the encryption dekGenerationId is initialized to 1 in the cache. It will not update to esxconf until it is bumped up. As a result, it is possible that dekGenerationId is not present in esx.conf in versions prior to 6.7U3.

In the process of upgrading ESXi hosts to 7.0, all configurations in esx.conf will be automatically migrated into a config store, and then the esx.conf file is discarded. If esx.conf does not contain the encryption dekGenerationId at that point, it cannot be migrated to config store. When the upgrade completes, from the perspective of ESX host, dekGenerationId is empty in configtore, but actually, encryption is enabled. This inconsistency triggers the vSAN health check. As a result, vSAN health check will report the vSAN cluster configuration consistency issue.

Resolution

The issue has been resolved in vSAN 7.0U3d 19482537

Workaround:

There are two ways to workaround this issue:

1. Trigger a cluster remediation by clicking the "REMEDIATE INCONSISTENT CONFIGURATION" button as seen in the above screenshot after upgrading to 7.0 version prior to 7.0U3d

2. Manually add dekGenerationId to the esx.conf file before upgrading to a version prior to 7.0U3d:

• Login to the ESX hosts.

• Check if dekGenerationId is stored at file /etc/vmware/esx.conf by below command.

>grep "dekGenerationId" /etc/vmware/esx.conf

• If no result from the above command, add below line into /etc/vmware/esx.conf

>/vsan/dekGenerationId = 1