"IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified", Registration/unregistration of third-party IO filter storage providers fails on a host

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• ESXi summary page shows the alarm "Registration/unregistration of third-party IO filter storage providers fails on a host" 
  
  
• Journalctl logs on vCenter Server bundle will show following the same failure events
  
  journalctl_-b--0.txt

[User logged event: Registration/unregistration of third-party IO filter storage providers fails.]
[User logged event: Registration/unregistration of third-party IO filter storage providers fails on a host on several ESXi hosts.]
  
  
• ESXi summary pages show alert "ESXi VASA client certificate provision has failed"
  
  
  
  
• Status of IOFilter for the respective hosts are shown as "Offline" under Storage Provider Tab (VC Root Object -> Configure -> Storage Providers)
  
  
  
  
• Iofiltervpd service logs on the ESXi repeatedly shows following error message while verifying the Certificate
  
  /var/run/log/iofiltervpd.log

YYYY-MM-DDTHH:MM:SSZ iofiltervpd[2098637]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified
YYYY-MM-DDTHH:MM:SSZ iofiltervpd[2098637]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified
YYYY-MM-DDTHH:MM:SSZ iofiltervpd[2098637]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified
YYYY-MM-DDTHH:MM:SSZ iofiltervpd[2098637]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified
YYYY-MM-DDTHH:MM:SSZ iofiltervpd[2098637]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified

Environment

vCenter Server 7.x
vCenter Server 8.x

 

Cause

This issue is caused when "sms_self_signed" certificate on the vCenter Server is missing on the ESXi host and due to that communication between Iofiltervpd service on the host and SPS (vmware-sps) service on the vCenter Server fails while verifying the certificate.

When Iofiltervpd service on the host connects to vCenter Server, it tries to verify the "sms_self_signed" certificate stored in SMS store on vCenter Server against the certificate stored in the file /etc/vmware/ssl/iofiltervp_castore.pem. This connection will fail if the "sms_self_signed" certificate is not available in the file iofiltervp_castore.pem on the host.

This can be verified using below steps:

1. Login to vCenter Server using SSH
2. Execute below command to list the "sms_self_signed" certificate
   
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store SMS --alias sms_self_signed
   
   
3. Login to ESXi host using SSH
4. List the file /etc/vmware/ssl/iofiltervp_castore.pem
   
   cat /etc/vmware/ssl/iofiltervp_castore.pem
   
   
5. Compare the certificate listed in Step 2 against the pem file from Step 4.
6. This issue will be observed it the Certificate from Step 2 is not available in the file from Step 4

Resolution

Manually copy the "sms_self_signed" certificate from vCenter Server to the file /etc/vmware/ssl/iofiltervp_castore.pem on the host.

• Login to vCenter Server using SSH
• Copy the certificate listed by below command
  
  /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store SMS --alias sms_self_signed
  
  Note: Copy the entire section including the "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----".
  
  
• Login to ESXi host using SSH
• Edit the file /etc/vmware/ssl/iofiltervp_castore.pem
  
  vi /etc/vmware/ssl/iofiltervp_castore.pem
  
  
• Paste the certificate copied from vCenter Server to this file (insert it at the beginning of the file)
• Save and exit the vi editor
• Restart the Iofiltervpd service on the host
  
  /etc/init.d/iofiltervpd restart
  
  
• Wait for 2 minutes and the IOFilter status will show as Online on vCenter Server.

Additional Information

ESXi VASA Client Certificate Provision Has Failed