Alarm for Registration/unregistration of third-party IO filter storage providers fails.

Products

• VMware SDDC Manager
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

• vCenter reports the warning "Registration/unregistration of third-party IO filter storage provider fails on a host."
• IO Filter entries are not shown within vCenter Storage Providers.
• In /var/log/iofiltervpd.log, you see an error similar to -

yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: IOFilterVPControlHandleMessage:239:received command: addconf /usr/lib/vmware/vmiof/disk/vmwarevmcrypt-config.xml VMW_vmwarevmcrypt_1.0.0
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:247:/usr/lib/vmware/vmiof/disk/vmwarevmcrypt-config.xml: successfully loaded filter configuration.
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:248:FilterName: vmwarevmcrypt
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:249:FilterClass: encryption
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:250:FilterVersion: 1.0.0
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:251:FilterType: disk
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: LoadFilterCatalogs:191:Added 1 catalog(s) for filter vmwarevmcrypt
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: LoadConfiguration:158:Added configuration file: /usr/lib/vmware/vmiof/disk/vmwarevmcrypt-config.xml successfully.
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: AddConfiguration:258:Number of filters loaded: 1
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: IOFilterVPControlHandleMessage:239:received command: addconf /usr/lib/vmware/vmiof/disk/spm-config.xml VMW_spm_1.0.0
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:247:/usr/lib/vmware/vmiof/disk/spm-config.xml: successfully loaded filter configuration.
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:248:FilterName: spm
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:249:FilterClass: datastoreIoControl
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:250:FilterVersion: 1.0.230
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: ReadConfigFromFile:251:FilterType: disk
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: LoadFilterCatalogs:191:Added 8 catalog(s) for filter spm
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: LoadConfiguration:158:Added configuration file: /usr/lib/vmware/vmiof/disk/spm-config.xml successfully.
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: AddConfiguration:258:Number of filters loaded: 2
yyyy-mm-ddThh:mm:ssZ iofiltervpd[67088]: run:159:SSL Connection error 30 : SSL_ERROR_SSL error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown

• Note: This can also be found in /var/run/log/ in the log bundle, as iofiltervpd.log is not present in /var/log
  
  
• Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Environment

VMware ESXi 7.0

VMware ESXi 8.0

Cause

The IO Filter on the ESXi host is not correctly registered with the ESXi host certificate.

Resolution

• To resolve this issue:- Refresh the IO Filter Registration on the ESXi Host:
   
  • Before refreshing the IO filter registration - Verify the Host Certificate and Certificate chain is OK on the ESXi. You can verify the certificate from vSphere client. Manage Host Certificates by Using the VMware Host Client
  • SSH to the ESXi host
  • Take a backup of the current cert file  /etc/vmware/ssl/castore.pem  ( by running: cp  /etc/vmware/ssl/castore.pem  /etc/vmware/ssl/castore.pem.backup)
  • Before refreshing the IO filter registration - Verify the Host Certificate and Certificate chain is ok on the ESXi - these are located at the following locations:
    /etc/vmware/ssl/rui.crt
    /etc/vmware/ssl/castore.pem 
  • Note: The following steps can be performed without placing the ESXi host into Maintenance Mode (MM). However, as a best practice and for added safety, it is recommended to carry out these steps while the ESXi host is in Maintenance Mode.
  • Manually refresh the certificates in the iofiltervp by running the following command:
    /usr/lib/vmware/iofilter/bin/iofvp-ctrl-app -r

Additional Information

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

• Product: ESXi
• Log File: iofiltervpd.log
• Log Expression Check "SSL_ERROR_SSL" AND "alert certificate unknown"