VVOL Sync Error failed to get endpoint, err=VP cannot be connected via url = https://<vasa_provier_uri_or_ip>:8084, using default after VC Machine_SSL and ESXi host Certificate replacement with Custom CA

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Running the following command shows unsupported certificate purpose/ unable to get local issuer certificate:

  • openssl verify -purpose sslclient -CAfile /etc/vmware/ssl/castore.pem /etc/vmware/ssl/rui.crt

    error 20 at 0 depth: unable to get local issuer certificate

• The advanced setting for vpxd.certmgmt.mode will be set to “custom” 
• After vCenter and/or ESXi host Certificate replacement, host losing connection to VVOL Datastore

  • VVOL shows syncError:

    esxcli storage vvol vasaprovider list

    vasa_provier_uri_or_ip

       VP Name: vp_name_uri_or_ip

       URL: https://vasa_provier_uri_or_ip:8084

       Status: syncError

       Arrays:

         Array Id: com.xxxxxxxxxx:########-####-####-####-########e4ec

          Is Active: true

          Priority: 200

• Checking /var/log/vvold.log, you see the below error:

  • YYYY-MM-DDTHH:MM:SS info vvold[2229026] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://<vasa_provier_uri_or_ip>:8084

    YYYY-MM-DDTHH:MM:SS warning vvold[2229026] [Originator@6876 sub=Default] status code=400,status message=Bad Request

    YYYY-MM-DDTHH:MM:SS warning vvold[2229026] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (vasa_provier_uri_or_ip)!

    YYYY-MM-DDTHH:MM:SS info vvold[2229026] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://vasa_provier_uri_or_ip:8084

    YYYY-MM-DDTHH:MM:SS error vvold[2229026] [Originator@6876 sub=Default] Initialize: Unable to init session to VP vasa_provier_uri_or_ip state: 0

    YYYY-MM-DDTHH:MM:SS info vvold[2229028] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://vasa_provier_uri_or_ip:8084

    YYYY-MM-DDTHH:MM:SS warning vvold[2229028] [Originator@6876 sub=Default] status code=400,status message=Bad Request

Environment

VMware vSphere ESXi 7.0

Cause

The issue occurs when the ESXi certificate does not have a correct key usage. The ESXi client certificate does not have Client Authentication and causes certificate validation to fail at the VASA Provider. Because ESXi is acting as a client to the VASA Provider,  Enhanced Key Usage (EKU) must contain Client Authentication

 

Resolution

• Run the below command from ESXi host to verify if there are any issues with the certificate:
  • openssl s_client -showcerts -cert /etc/vmware/ssl/rui.crt -key /etc/vmware/ssl/rui.key -CAfile /etc/vmware/ssl/castore.pem -connect <VP_IP_FQDN>:PORT
• Check the ESXi host certificate and verify Enhanced Key Usage contains both server and client authentication:
  • 
• If the custom certificate on ESXi host has Server Authentication but not Client Authentication in the EKU, it needs to be replaced so that it contains both.
•  Using openssl verify command and ensure no issues are observed:

  • openssl verify -purpose sslclient -CAfile /etc/vmware/ssl/castore.pem /etc/vmware/ssl/rui.crt

    Expected result:

    /etc/vmware/ssl/rui.crt: OK