When creating VSAN ESA cluster we get the error: "Key provider NW-NKP is not available on host."

Products

• VMware vSAN

Issue/Introduction

When creating VSAN ESA cluster we get the error: "Key provider NW-NKP is not available on host.": 

When you see this, go ahead and check the vSAN health log here:  /var/log/vmware/vsan-health/vmware-vsan-health-service.log. If you see the following message then we know that we need to renew the solution user certificate on vSAN: 

faultcode: ns0:FailedAuthentication

faultstring: Invalid credentials

faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.

org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

YYYY-MM-DDTHH:MM:SS.938-06:00 ERROR vsan-mgmt[1803132] [VsanVapiUtil::GetVapiConfigStubBySolUser opID=agw-0004339-6f08] Fail to connect vAPI by solution user vpxd-extension

Traceback (most recent call last):

  File "bora/vsan/health/vpxd/pyMoVsan/VsanVapiUtil.py", line 161, in GetVapiConfigStubBySolUser

  File "bora/vsan/health/vpxd/pyMoVsan/VsanVapiUtil.py", line 140, in _getConfigStubBySolUser

  File "bora/vsan/health/vpxd/pyMoVsan/VsanVapiUtil.py", line 103, in _getSamlToken

  File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 388, in get_hok_saml_assertion

  File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 277, in perform_request

pyVim.sso.SoapException: SoapException:

faultcode: ns0:FailedAuthentication

faultstring: Invalid credentials

Environment

VMware vSAN 9.0.x

VMware vSAN 8.0.x

VMware vSAN 7.0.x

 

Cause

This is caused by an invalid solution user certificate on vCenter. 

Resolution

To resolve this, you need to renew the solution user certificate. To do so, you have two options; 

1. Use the built in vCenter Certificate Manager
   • To use this utility you will select one of the two options depending on if you use self signed solution user certificates. 
     1. If you have a custom CA cert select: 5. "Replace the Solution User Certificates with Custom CA Certificates"
     2. If you are using VMCA signed certs then select 6. "Replace the Solution User Certificates with VMCA generated Certificates."   
2. Use the vCert scripted certificate replacement tool
   • If you select these options when you run the vCert Tool: 
     1. Select option 3. "Manage certificates"
     2. Then select option 2. "Solution User certificates"
        • This option replaces the Solution User certificates in VECS and updates the Service Principal entries in VMware Directory. The vpxd-extension thumbprints are updated in vCenter database. A VMCA-signed certificate or custom CA-signed certificate can be used.
          
          • Custom CA-signed certificates - There is an option to generate a private key and Certificate Signing Request or import the signed certificate and key. If the presented CA-signed certificate does not include a complete CA chain then the script will prompt for a file containing the complete chain.