Firstboot error while VC deployment in vSAN encrypted datastore

Products

• VMware vCenter Server

Issue/Introduction

• vCenter deployment failed with the first boot error as below:
  
  VMware vSAN Health service failed firstboot.
  
  
  
  
• The environment has been previously using a Native Key Provider (NKP) for datastore encryption and the vCenter is been redeployed.
  
  
• vCenter logs reporting the following errors:
  
  /var/log/firstboot/vsanhealth_firstboot.py_xxxxx_stderr.log
Traceback (most recent call last):
  File "/usr/lib/vmware-vpx/firstboot/vsanhealth_firstboot.py", line 418, in Main
    waitServiceAvailableThenDoVsanBootstrap()
  File "/usr/lib/vmware-vpx/firstboot/vsanhealth_firstboot.py", line 175, in waitServiceAvailableThenDoVsanBootstrap
    raise ex
  File "/usr/lib/vmware-vpx/firstboot/vsanhealth_firstboot.py", line 168, in waitServiceAvailableThenDoVsanBootstrap
    doVsanBootstrap()
  File "/usr/lib/vmware-vpx/firstboot/vsanhealth_firstboot.py", line 191, in doVsanBootstrap
    bootstrap.postDeployConfig(setting)
  File "/usr/lib/vmware-vpx/firstboot/vsanhealth_firstboot.py", line 155, in postDeployConfig
    raise Exception("Vsan Post Deploy Config Failed")
Exception: Vsan Post Deploy Config Failed
xxxx-xx-xxTxx:xx:xx.xxxZ  VSAN Health firstboot failed

/var/log/vmware/vsan-health/vmware-vsan-health-service.log
xxxx-xx-xxTxx:xx:xx.xxxZ ERROR vsan-mgmt[xxxxx] [VsanVcExtension::_Connect opID=noOpId] Get vpxd connection error (vim.fault.InvalidLogin) {
xxxx-xx-xxTxx:xx:xx.xxxZ ERROR vsan-mgmt[xxxxx] [VsanVcClusterUtil::FindAllVcClusters opID=noOpId] Fail to get vSAN enabled cluster : ('vsan-cluster-mgmt-internal-system', None)
xxxx-xx-xxTxx:xx:xx.xxxZ ERROR vsan-mgmt[xxxxx] [EncryptPass::RefreshEncryptKey opID=noOpId] Failed to refresh encrypt key file
xxxx-xx-xxTxx:xx:xx.xxxZ ERROR vsan-mgmt[xxxxx] [VsanVcsaDeployerSystemImpl::BootstrapPostConfig opID=noOpId] Got exception: No native key provider spec provided
Traceback (most recent call last):
  File "bora/vsan/health/esx/pyMo/VsanVcsaDeployerSystemImpl.py", line 750, in BootstrapPostConfig
Exception: No native key provider spec provided

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

The environment was previously encrypted using a Native Key provider.

Resolution

To resolve the issue recover the missing NKP following the steps from the documentation below: 

Recovering a vSphere Native Key Provider

Additional Information

vSphere Native Key Provider Overview
vSphere Native Key Provider (NKP) Questions & Answers
Enable Encryption on a New vSAN Cluster