IO Filters marked offline in a vSAN cluster due to missing Self Signed SSL certificate in /etc/vmware/ssl/castore.pem

Products

• VMware vCenter Server

Issue/Introduction

I/O Filters are marked offline in a vSAN cluster. One of the reasons can be they are missing Self Signed SSL certificate in /etc/vmware/ssl/castore.pem

• The vSAN GUI (vCenter > Configure > Storage Providers) may show all the hosts I/Ofilter storage providers as:
  
  offline


• The Re-scan or re-synchronize of the VASA providers does not make any change to the state of the I/Ofilter.  An upgrade of ESXi does not resolve the issue.
  
  
• You may find the below instances showing that SSL certs are not being verified for the hosts in the IOfiltervpd logs (ESXi : /var/log/iofiltervpd.log)
  

  2019-03-07T04:12:50Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can't be verified
  2019-03-07T04:13:00Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can't be verified
  2019-03-07T04:13:00Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can't be verified
  2019-03-07T04:13:10Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can't be verified
  2019-03-07T04:13:20Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can't be verified

• The newly added host may show the I/Ofilters proviiders as:
  online
• You may find that the certificates located at /etc/vmware/ssl/castore.pem, have the missing "Self Signed Certificate" for the host with IOfilters offline
  
  Below is an example Self signed certificate which must be part of the /etc/vmware/ssl/castore.pem file .
  
  Example of "Self signed certificate" :
  

  -----BEGIN CERTIFICATE-----
  MIIB1zCCAUCgAwIBAgIGAVdts0########QEBBQUAMC8xDzANBgNVBAoT
  ##############################################
  Ar+mt3CKX11Z8F+qWg20lvwRh#########XXYrxUbyYPkrSaOdrw==
  -----END CERTIFICATE-----

Note: Please validate if a self signed certificate or a custom certificate is being used. Custom certificate information can be found in the "additional information" section of this article

Cause

The default certificates are required for the correct communication. When these certificates are missing, communication cannot be established because there is no verification and validation of who the conversation is taking place with.

Resolution

1. Put the host in Maintenance mode with Ensure Accessibility.
   
   
2. Take a backup of the current cert file /etc/vmware/ssl/castore.pem.
   
   Example:
   cp /etc/vmware/ssl/castore.pem /etc/vmware/ssl/castore.pem.backup


3. Copy the file /etc/vmware/ssl/castore.pem from a working host to the affected hosts (Alternatively you can use scp or winscp to upload a copy of the file directly to /etc/vmware/ssl/)
   
   
4. Replace the older file with newer one.
   
   Example:
   cp /tmp/castore.pem /etc/vmware/ssl/castore.pem


5. Rebooted the hosts one at a time.
   
   
6. After the reboot the providers should show as "online" in vCenter > Configure > Storage Providers.

Additional Information

Adding Custom Certificate on ESXi hosts through CLI