Alarm for Registration/un-registration of third-party IO filter storage providers fails

Products

VMware SDDC Manager / VCF Installer VMware vCenter Server 7.0 VMware vCenter Server 8.0 VMware vSphere ESXi

Issue/Introduction

vCenter reports a warning:
Registration/un-registration of third-party IO filter storage provider fails on a host.
Reviewing the storage providers registered with the vCenter Server using vSphere Client (vCenter > Configure > Storage Providers), there is no IOFILTER Provider listed for the host.

When reviewing the IO filter daemon log of the ESX/ESXi host, /var/run/log/iofiltervpd.log, entries similar to the following ones can be found:

yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: IOFilterVPControlHandleMessage:239:received command: addconf /usr/lib/vmware/vmiof/disk/vmwarevmcrypt-config.xml VMW_vmwarevmcrypt_1.0.0
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:247:/usr/lib/vmware/vmiof/disk/vmwarevmcrypt-config.xml: successfully loaded filter configuration.
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:248:FilterName: vmwarevmcrypt
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:249:FilterClass: encryption
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:250:FilterVersion: 1.0.0
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:251:FilterType: disk
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: LoadFilterCatalogs:191:Added 1 catalog(s) for filter vmwarevmcrypt
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: LoadConfiguration:158:Added configuration file: /usr/lib/vmware/vmiof/disk/vmwarevmcrypt-config.xml successfully.
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: AddConfiguration:258:Number of filters loaded: 1
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: IOFilterVPControlHandleMessage:239:received command: addconf /usr/lib/vmware/vmiof/disk/spm-config.xml VMW_spm_1.0.0
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:247:/usr/lib/vmware/vmiof/disk/spm-config.xml: successfully loaded filter configuration.
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:248:FilterName: spm
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:249:FilterClass: datastoreIoControl
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:250:FilterVersion: 1.0.230
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: ReadConfigFromFile:251:FilterType: disk
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: LoadFilterCatalogs:191:Added 8 catalog(s) for filter spm
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: LoadConfiguration:158:Added configuration file: /usr/lib/vmware/vmiof/disk/spm-config.xml successfully.
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: AddConfiguration:258:Number of filters loaded: 2
yyyy-mm-ddThh:mm:ssZ iofiltervpd[###]: run:159:SSL Connection error 30 : SSL_ERROR_SSL error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

Environment

VMware ESXi 7.0.x
VMware ESXi 8.0.x
VMware ESX 9.x

Cause

This issue occurs when the IO Filter on the ESXi/ESX is not correctly registered with the ESXi/ESX host SSL certificate.
It can also occur when moving hosts between Datacenters without setting them into maintenance mode first.

Resolution

Note: The following steps can be performed without placing the ESXi/ESX host into Maintenance Mode (MM). However, as a best practice and for added safety, it is recommended to carry out these steps while the ESXi host is in Maintenance Mode.

To fix this issue, refresh the IO Filter registration on the ESXi/ESX Host:

Before refreshing the IO filter registration, verify the Host Certificate and Certificate chain is OK on the ESXi/ESX. To verify the certificate use the vSphere client. Refer to Manage Host Certificates by Using the VMware Host Client for more information.
Connect to the specific ESXi/ESX host per SSH and login as the root user.
The current certificate files can be found in the following locations:

File Content
/etc/vmware/ssl/rui.crt This file contains the actual SSL host certificate.
/etc/vmware/ssl/castore.pem This is the trusted certificate store of the ESXi host.

File	Content
`/etc/vmware/ssl/rui.crt`	This file contains the actual SSL host certificate.
`/etc/vmware/ssl/castore.pem`	This is the trusted certificate store of the ESXi host.

Take a backup of the trusted certificate store castore.pem:

# cp /etc/vmware/ssl/castore.pem /etc/vmware/ssl/castore.pem.backup

Depending on the version of the host, run either of the following commands to manually refresh the certificates for iofiltervpd:
1. For ESXi 7.0.x or 8.0.x, run:
```
# /usr/lib/vmware/iofilter/bin/iofvp-ctrl-app -r
```
2. For ESX 9.x, run:
```
# /etc/init.d/hostd restart && /etc/init.d/vpxa restart
```
After performing these steps, right click the alarm in vSphere Client, acknowledge it and reset the status to "Green".