VSAN Health warning "Failed to get host encryption health result" after certificate renewal/replacement in vCenter or ESXi hosts

Products

• VMware vCenter Server 8.0
• VMware vSAN 8.x

Issue/Introduction

• Attempts at a rekey (Generate new Encryption Keys) fail with the current Native Key Provider.
  
  
• Creating a new NKP also fails with "Key provider XXXXXX is not available on host".
  
  
• This Issue may be reported even when ESXi certificates are renewed / replaced.
  
  
• Below are the log snippets reported at the time of the issue:

- vmware-vsan-health-service log (/var/log/vmware/vsan-health/vmware-vsan-health-service-XXX.log) in vCenter server reports:

faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>



- /var/log/vmware/vsan-health/vmware-vsan-health-summary-result.log reports Failed to Get Host Encryption Health Result error:

[YYYY-MM-DDTHH:MM] INFO vsan-mgmt[2700617] [VsanHealthSummaryLogUtil::PrintHealthResult opID=noOpId] Cluster <Cluster-name>  Overall Health : yellow
   Group encryption health : yellow
      Test kmsconnection health : yellow
         HostsKmsStatus: Hosts  KmsCluster  KmsAlias  ConnectionStatus  KeyState  Issue  Recommendation
                         (Host-XXX1, Nkp-XXX, '', Yellow, Yellow, FailedToGetHostEncryptionHealthResult, ''), (Host-XXX1, Nkp-XXX, '', Yellow, Yellow, FailedToGetHostEncryptionHealthResult, ''),
                         (Host-XXX2, Nkp-XXX, '', Yellow, Yellow, FailedToGetHostEncryptionHealthResult, ''), (Host-XXX2, Nkp-XXX, '', Yellow, Yellow, FailedToGetHostEncryptionHealthResult,''),
                         (Host-XXX3, Nkp-XXX, '', Yellow, Yellow, FailedToGetHostEncryptionHealthResult, ''), (Host-XXX3, Nkp-XXX, '', Yellow, Yellow, FailedToGetHostEncryptionHealthResult, ''),

      Test hostcpuaesni health : unknown
         HostsCpuAes-NiState: Hosts  Status  Reason

                              (Host-XXX1, Unknown, Com.Vmware.Vsan.Health.Test.Cloudhealth.Hostcpuaesni.Testresult.Hostcloudhealthresulterror), (Host-XXX1, Unknown, Com.Vmware.Vsan.Health.Test.Cloudhealth.Hostcpuaesni.Testresult.Hostcloudhealthresulterror),
                              (Host-XXX2, Unknown, Com.Vmware.Vsan.Health.Test.Cloudhealth.Hostcpuaesni.Testresult.Hostcloudhealthresulterror), (Host-XXX2, Unknown, Com.Vmware.Vsan.Health.Test.Cloudhealth.Hostcpuaesni.Testresult.Hostcloudhealthresulterror),
                              (Host-XXX3, Unknown, Com.Vmware.Vsan.Health.Test.Cloudhealth.Hostcpuaesni.Testresult.Hostcloudhealthresulterror), (Host-XXX3, Unknown, Com.Vmware.Vsan.Health.Test.Cloudhealth.Hostcpuaesni.Testresult.Hostcloudhealthresulterror),

- vsanvcmgmtd.log reports Not Authenticated errors as follow:

[YYYY-MM-DDTHH:MM] info vsanvcmgmtd[09344] [vSAN@6876 sub=vmomi.soapStub[4] opId=##########] SOAP request returned HTTP failure; <<cs p::##############, TCP:localhost:####>, /ls/sdk>, method: searchNotifications; code: 500(Internal Server Error); fault: (cis.license.fault.NotAuthenticatedFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<cs p:##############, TCP:localhost:####>, /ls/sdk>]: searchNotifications
--> Authentication result: Saml token expired!"
--> }


- ESXi host vsanmgmt log (/var/run/log/vsanmgmt.log):

[YYYY-MM-DDTHH:MM] ERROR vsan-mgmt[33925] [VsanVapiUtil::GetVapiConfigStubBySolUser opID=agw-00xxxxx-xxxx] Fail to connect vAPI by solution user vpxd-extension
Traceback (most recent call last):
File "bora/vsan/health/vpxd/pyMoVsan/VsanVapiUtil.py", line 161, in GetVapiConfigStubBySolUser
File "bora/vsan/health/vpxd/pyMoVsan/VsanVapiUtil.py", line 140, in _getConfigStubBySolUser
File "bora/vsan/health/vpxd/pyMoVsan/VsanVapiUtil.py", line 103, in _getSamlToken
File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 388, in get_hok_saml_assertion
File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 277, in perform_request
pyVim.sso.SoapException: SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Invalid credentials

Environment

• vCenter Server 8.0 U3
  
  
• VSAN 8.0 U3

Cause

vpxd-extension cert and key in vCenter Server at /storage/vsan-health does not get updated after vCenter or ESXi host certificate replacement.

Resolution

• Upgrade vCenter/ESXi to 8.0U3e that has fix for this issue.
  
  
• Workaround:
  
  
• • Take backup of the vCenter Server Appliance. If the vCenter Sever is in linked mode, refer KB: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice 
    
    
  • Connect SSH to the vCenter Server.
    
    
  • Backup vpxd-extension.cert and vpxd-extension.key located in /storage/vsan-health.

cp /storage/vsan-health/vpxd-extension.cert /storage/vsan-health/vpxd-extension.cert.bak
cp /storage/vsan-health/vpxd-extension.key /storage/vsan-health/vpxd-extension.key.bak

• • Remove vpxd-extension.cert and vpxd-extension.key from /storage/vsan-health.

rm /storage/vsan-health/vpxd-extension.cert
rm /storage/vsan-health/vpxd-extension.key

• • Copy the vpxd-extension certificate and key from the vecs store to /storage/vsan-health location.

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension > /storage/vsan-health/vpxd-extension.cert
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension > /storage/vsan-health/vpxd-extension.key

• • Restart vsan-health service.

vmon-cli -r vsan-health