VSAN Health warning "Failed to get host encryption health result" after resetting all certificates with VCenter Certificate Manager, Option 8

Products

• VMware vCenter Server 8.0
• VMware vSAN 8.x

Issue/Introduction

• Attempts at a rekey (Generate new Encryption Keys) fail with the current Native Key Provider.
• Creating a new NKP also fails with "Key provider XXXXXX is not available on host".
• Below are the log snippets reported at the time of the issue:

vCenter Server vmware-vsan-health-service log (/var/log/vmware/vsan-health/vmware-vsan-health-service-XXX.log):

faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

ESXi host vsanmgmt log (/var/run/log/vsanmgmt.log):

[YYYY-MM-DDTHH:MM] ERROR vsan-mgmt[33925] [VsanVapiUtil::GetVapiConfigStubBySolUser opID=agw-00xxxxx-xxxx] Fail to connect vAPI by solution user vpxd-extension
Traceback (most recent call last):
File "bora/vsan/health/vpxd/pyMoVsan/VsanVapiUtil.py", line 161, in GetVapiConfigStubBySolUser
File "bora/vsan/health/vpxd/pyMoVsan/VsanVapiUtil.py", line 140, in _getConfigStubBySolUser
File "bora/vsan/health/vpxd/pyMoVsan/VsanVapiUtil.py", line 103, in _getSamlToken
File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 388, in get_hok_saml_assertion
File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 277, in perform_request
pyVim.sso.SoapException: SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Invalid credentials

Environment

• vCenter Server 8.0 U3
• VSAN 8.0 U3

Cause

In a rare instance, the copy of vpxd-extension cert and key in the vCenter Server at /storage/vsan-health does not get updated after running Option 8 in certificate-manager.

Resolution

Workaround:

• Take backup of the vCenter Server Appliance. If the vCenter Sever is in linked mode, refer KB: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice 
• Connect SSH to the vCenter Server.
• Backup vpxd-extension.cert and vpxd-extension.key located in /storage/vsan-health.

cp /storage/vsan-health/vpxd-extension.cert /storage/vsan-health/vpxd-extension.cert.bak
cp /storage/vsan-health/vpxd-extension.key /storage/vsan-health/vpxd-extension.key.bak

• Remove vpxd-extension.cert and vpxd-extension.key from /storage/vsan-health.

rm /storage/vsan-health/vpxd-extension.cert
rm /storage/vsan-health/vpxd-extension.key

• Copy the vpxd-extension certificate and key from the vecs store to /storage/vsan-health location.

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension > /storage/vsan-health/vpxd-extension.cert
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension > /storage/vsan-health/vpxd-extension.key

• Restart vsan-health service.

vmon-cli -r vsan-health